UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

A Recognition Method for
Combined Activities withAccelerometers
Abstract

Kazuya Murao
Ritsumeikan University
1-1-1 Nojihigashi, Kusatsu,
Shiga, 5258577 Japanmurao@cs.ritsumei.ac.jp

Many activity recognition systems using accelerometers
have been proposed. Activities that have been recognized
are “single” activities which can be expressed with one
verb, such as sitting, walking, holding a mobile phone, andthrowing a ball. In actual, however, “combined” activities
including more than two kinds of state and movement areoften taken place. Focusing on hand gestures, they are
performed not only while standing, but also while walking
and sitting. Though the simplest way to recognize suchcombined activities is to construct the recognition models
for all the possible combinations of the activities, thenumber of combinations becomes immense. In this paper,
we propose a recognition method for combined activities
by learning single activities only. Evaluation results
confirmed that our proposed method achieved 0.84 ofrecall and 0.85 of precision, which is comparable to themethod that had learned all the combined activities.

Tsutomu Terada
Kobe University
1-1 Rokkodaicho, Nada, Kobe
Hyogo, 6578501 Japantsutomu@eedept.kobe-u.ac.jp

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
UbiComp ’14, September 13 - 17 2014, Seattle, WA, USA
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-3047-3/14/09...$15.00.
http://dx.doi.org/10.1145/2638728.2641304

787

Author Keywords
Activity recognition, Combined activity, Accelerometer

ACM Classification Keywords
H.4 [Information Systems Applications]: Miscellaneous

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Introduction
Along with the progress in wearable computing, many
context-aware systems with accelerometers have recently
been introduced. In the procedure of making an activity
recognition system, developers generally define theobjective activities, collect their data, annotate them, andconstruct the recognition models. Therefore, recognition
results are limited within the predefined activities. In
other word, if we want to recognize a combined activity
holding a mobile phone while walking, training data forthe activity has to be collected and annotated so,otherwise, recognition result would be either walking or
holding a mobile phone.
Here, we define two types of activities; global activity andlocal activity. Global activity is a bodywide movement,such as walking and standing. Multiple global activities
contradict each other at a time. Local activity is amovement of a specific parts of body, such as throwing
and holding something. Multiple local activities coexist
unless these are on the same part. The simplest way torecognize combined activity is to construct recognition
models for all the combined activities, but the number ofthe possible combinations of global and local activities
gets immense. Supposing five global activities; standing,sitting, walking, running, and bicycling, and ten local
activities of hand gestures, data for the 50 patters must
be collected. As one kind of hand gesture is added, data
for the gesture performed during five global activities must
be captured, and it is a backbreaking task. Moreover,considering foot gestures, the possible patters are global
activity×local hand activity×local foot activity. However,these combined activities are not negligible since they arephysically possible and might occur in our daily life.
Problem of combined-activity recognition is not only

788

taking time for collecting training data, but alsoannotating data. After collecting all the data throughlong experiment, the data have to be annotated withground truth. Figure 1 shows acceleration of a chopgesture while running and while standing. It is easy tofind starting point and end point for chop while standing,whereas it is hard to trim it exactly by visual check forchop while running. Someone might think thatrecognizing activity of each part of body individually andintegrating them produce correct activity. However, as
shown in Figure 1, a gesture while running is diﬀerent tothat while standing. Moreover, arm swing of running is
included in the beginning and ending of the gesture unlessthe gesture is trimmed with a tightfitting window. Motionof running is propagated, the gesturing is slightly
diﬀerent, and part of running motion is included before
and after the gesture, which leads to misrecognition.

End point

Starting pointand end point
are not clear.

R arm xR arm yR arm z
Running
Chop

Starting point

R arm xR arm yR arm zStanding
Chop

Figure 1: Waveform of an accelerometer mounted on the rightwrist: a chop gesture while running (left) and while standing(right).

Our previous work tackled on gesture recognition whilemoving. Gestures having been recognized in the research
are not combined ones, but single ones which aresmoothly changed from and to global activity, such as
walking[3]. An eﬀective method to treat combined
activity has not been reported.
We developed the system that recognizes combined

WORKSHOP: HASCA

activities from training data of single activities, i.e. global
activities and local activities while standing. The systemclassifies each part of the body to posture, behavior, andgesture, from fluctuation and autocorrelation in theacceleration data. In this work, we define posture,
behavior, and gesture as follows: Posture is a state of auser remaining stationary lasting for a certain length oftime, e.g. sitting and standing. Behavior is a state of auser doing periodical movement lasting for a certain
length of time, e.g. walking and running. Gesture is not astate but once-oﬀ action having starting point andendpoint that sporadically occur, e.g. punch and draw acircle in the air.
In general, autocorrelation plot of a periodical wave showshigh peaks. When parts of sensors are showing highpeaks, the corresponding body parts are meant to haveconstancy and are classified to behavior, otherwise theparts are classified to gesture. Finally, the system outputs
conclusive recognition result from the recognition resultsof each part. By using our system, combined activities
such as throw while walking and holding a mobile phone
while running are recognized only from single activities;
walking, running, throwing, and holding a mobile phone.

Related work
Studies on activity recognition are listed in Table 1,
however most of them focus on single activities, such as
ambulation and posture. One study recognizes eightactivities including vacuuming and brushing teeth with anaccelerometer attached to the pelvic region[4]. Other
study employs twenty-two kinds of sensors to recognize
lying, rowing, running, Nordic walking, bike, walking,sitting, and standing[1]. Though walking carrying items
activity seems combined activity, it cannot be separated
and recognized by combining other activities, i.e. runningcarrying items activity and holding items activity cannot
be recognized in this architecture.
Table 1: Activities recognized and sensors used in past workon activity recognition.
Ref.

Activities recognized[1]

ambulation, posture,scrubbing, vacuuming,folding laundry, brushing teeth,
bicycling, eating or drinking,working on computer,walking carrying items
ambulation,
posture,putting clothes,eating,rowing,bicycling,
respiratory

[2]

[4]
[5]
[6]

[8]

# of
sensor5

ambulation,posture, sit-ups,
vacuuming, brushing teeth
hammer, file, sand, saw, screw, vise,
drill, clap, use driver, grind
push button, handshake,phone up, phone down, cutlery,door, coin, drink, spoon, handheld
draw ‘>’ mark, square
shift left to right,
shift right to left,
shift bottom to up,
shift up to bottom,
clockwise circle,
anticlockwise circle

22

2D acc×5

Sensorpositionleft elbow
right wrist
torso
left kneeright ankle

1

air pressure, mic, pulse,ball sensor, light, switch,
EKG, humidity, 3D acc×2,
temperature, respiratory,
3D compass×2, GPS,heart rate×3, SaO2×2,
skin temperature,skin resistance
acc

wrist
upper backbelow neckfingerarmpitchest
foreheadshoulderpelvic region

2

acc×2

5

gyro×51

wii remote

wrist
elbow
wristsupper arms
upper torso
handGeorgia Tech Gesture Toolkit[5] as a tool to support
gesture recognition has been proposed by Westeyn et al.
This is a toolkit that enables ordinary users who do nothave enough knowledge on speech recognition to use
existing HMMtoolkit1 with ease. In the literature, four
applications are presented, one is gesture recognition withtwo 3-axis accelerometers positioned at wrist and elbow,
and achieves 93.3% accuracy for ten kinds of gestures
such as grinding, sawing, and screwing. The system
proposed by Junker et al.[6] recognizes ten daily shortactions, such as pushing a button and drinking, andachieves approximately 80% precision and recall.
1 HTK

HiddenMarkovModel

http://htk.eng.kcam.ac.uk/.

789

Sensor kind

Toolkit

home

page.

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Innovative point of this study is that it partitions thestream of sensor into several segments that represent
atomic human movement by using the sliding-window andbottom-up (SWAB) algorithm[7]. The method proposedby Liu et al. recognizes eight gestures such as drawing aline and a circle, which are recommended by Nokia
laboratory, with one 3-axis accelerometer[8]. This research
captures more than 4,000 samples from eight test subjects
for a long period. They use Dynamic Time Warping
algorithm (DTW)[9] as a recognition algorithm andachieve 98.6% accuracy by successively renewing training
data.
Activities that have been targeted in these works are single
activities. Combined activities have to be defined one byone. However, the number of possible combinations
increases in multiple order of the number of global andlocal activities, causing much time to capture training
data. Capturing ten global activities and ten 10-second
local gestures for five times takes 5,000 seconds. This isan actual time of movement and more than double or
triple time is needed considering interval and rest.

System structure
Posture and behavior are state lasting for a certain periodof time and consist of periodic pattern of acceleration
waveform. These activities are generally recognized with aclassifier such as SVM[13] after converting raw data into
mean, variance, fast Fourier transform (FFT) coeﬃcient
over a time window. This approach enables high-speed
recognition since not all data in the window but feature
value is used. Moreover, one of the advantages of using
feature value is also that the recognition process does nothave to consider which part of the movement is includedin the window, e.g. beginning of the window does nothave to fit the specific motion of steps of walking, since790

feature value discards temporal information.
On the other hand, gesture is a once-oﬀ action that has
starting point and endpoint, which is diﬀerent to postureand behavior. Feature-based approach cannot distinguishsimilar gestures such as rotating arm clockwise andanticlockwise since feature value does not haveinformation on how it moved, therefore gesture must berecognized in a diﬀerent way. In general, gestures arerecognized with a template matching algorithm such as
dynamic time warping (DTW) or a statistical model suchas hidden Markov models (HMM), after trimming an
actual movement from stream data. The conventional
method forced the users to indicate gesture interval bypushing a button of a device or by standing still before
and after the gesture[10]. It is hard to stop motions or
perform specific gestures to indicate starting point. Taking
out a device or keep holding a device to push button while
performing gestures is also unrealistic. Recognizing a
gesture without considering starting point and endpoint,
misrecognition occurs or miss out the gesture buried inbehavior. Our system classifies user activities into posture,
behavior, and gesture for each body part, then apply
DTW to gesture or apply SVM to posture and behavior.
Our system consists of three phases, as shown in Figure 2.
The first phase classifies activities of each part of body
into three types; posture, behavior, and gesture. The
second phase recognizes activities according to theactivity type. The third phase integrates the recognition
results and outputs conclusive result.
In this paper, we assume that user attaches fiveaccelerometers on their both wrist, hip, and both ankle.
Activities are four postures (sitting, standing, lying, andkneeling), five behaviors (walking, running, bicycling,
descending stairs, and ascending stairs), five

WORKSHOP: HASCA

hand-gestures (chop, throw, punch, draw a clockwise
circle, and draw a anticlockwise circle), and two
hand-postures (holding a mobile phone, and raising ahand). The sampling frequency is 20 [Hz], which is
suﬃcient for activity recognition as reported in [11].
Activityclassification

Activityrecognition

Right arm
Left arm
Hip
Right leg
Left leg
User

GestureChop

Behavior

Walk

Behavior

Walk

BehaviorRun

Behavior

Walk

Activity
integration

Chop withwalking

was up to 100 [mG]. Since the current value x(t) might
temporarily goes into the epsilon tube even while moving,
posture begins only after x(t) stays within the epsilon
tube for more than 0.25 second. These values are
obtained from our pilot studies. As shown in Figure 3,
while the data is within the epsilon tube, the systemjudges that the parts of body are maintaining a posture.
When the data indicates movement, this process goes on
to constancy decision phase.
Raw data x(t)WalkGlobal activity
by weighted
majory decision+εMoving average x(t)-εSensor data

Figure 2: Recognition flow.

Activity classification
Displacement detection
Activity classification phase checks for displacements inthe sensed data. Supposing time t = T now, if a
diﬀerence of moving average over 20-sample (1-second)sensed data x(T ) and current value x(T ) runs over a
threshold ε, our system detects a movement as a following
equation. Otherwise, our system judges that the user ismaintaining a posture.
if |x(T) − x(T)| > ϵotherwise⇒⇒

Behavior or Gesture
Posture(1)

The region of x(t) ± ε is called the epsilon tube, whichremoves displacements. In this paper,
ε = max{ave{std(Leg)},200}, where√
Leg = leg(x)2 + leg(y)2 + leg(z)2 [mG]. std is astandard deviation over a window and ave is an averageover both legs. The vibration of movement of legs is
propagated to hand, which produces not-small values.
While the movement of legs is not intense, ε is set to 200[mG] since fluctuation produced while being stationary

791

Posturet
Behavior or gesture(Go to constancy decision)

Figure 3: Displacement detection.

Constancy decision
Basically, data on walking include iterations in rhythmwith the steps. On the other hand, gestures are once-oﬀ
actions and do not have iterations. Note that we consider
that the iterations of once-oﬀ actions are behaviors. In
this phase, the autocorrelation function (ACF) finds
iterations in the user’s movements, and classifies the
movement into behavior or gesture. The discrete ACF
Rxx(τ ) at lag τ for a data sequence x(t) is defined as∑N −1
Rxx(τ ) = t=0 x(t)x(t − τ ), where N is the window
size for the ACF calculation and set to 64 samples (3.2
seconds), which is long enough to capture at least two
iterations. In addition, since ACF shows a maximum at τ
= 0, all the values of ACF are normalized by
R′ xx(τ ) = Rxx(τ )/Rxx(0) so that the range is(−1, +1). The system has to decide whether the

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

movement has constancy or not. Figure 4 shows
acceleration waveform and its ACF of walk, chop, thenwalk again activity. As shown in the figure, the ACF of
walking shows clear peaks, whereas the ACF of chop does
not have high peaks. Constancy is detected when theheight of the first peak R′ xx(n)(n > 0) exceeds
α · (1 − n/N ), where α is a coeﬃcient set to 0.6. Reason
n/N is used is that the height of the first peak linearlydecreases as τ increases.

2000

(2)

1

Peaks appear

0.5

1000
0

0

-0.5

-1000
-2000

Autocorrelation

Acceleration [mG]

⇒ Behavior⇒ Gesture
Autocorrelation

if R′xx (n) ≥ α · (1 − n/N)
otherwise

0

10

20

30

40
Lag τ

0

10

20

30

40
Lag τ

0.5

-3000
1250 1300 1350 1400

1450 1500 1550 1600
Time [1/20 second]

1650 1700

50

60

70

60

70

1

No peak

0

-0.5

50

Figure 4: Accelerations of chop gesture while walking (left)
and autocorrelation of walking (upper right) and chop (lower
right).

Intille et al. focus on acquiring in-situ training data andmention that acceleration data of walking in laboratory
displays consistent gait cycle. On the other hand,acceleration data of the same person outside thelaboratory may display marked fluctuation in gait cycle
and length[12]. Also we think that is true and the dataobtained from same subject on diﬀerent days are diﬀerent.
Although acceleration data fluctuate in the range of day
or hour, data in the range of few seconds or shorter are
significantly periodic, which could produce constancy.
Moreover, since our approach is unsupervised and does
not require training, influence of diﬀerences amongindividuals and users’ conditions are small.

792

Activity recognition
Activity recognition phase recognizes activities of eachpart of body according to the activity types. For posturedata, the mean value of the data in the window is
calculated as a feature value and the posture is recognizedwith SVM[13] that has learned only postures. Since thevariance of postures is almost zero, only the mean is used
for the recognition. SVM operating on the mean andvariance is used for behavior data, whereas DTW[9]operating on trimmed original wave over a window is used
for recognizing gestures. SVM and DTW has learned
behaviors only and gestures only, respectively.
Activity integration
Since each part of the body outputs activity recognition
result individually, these have to be integrated in order todecide conclusive recognition result. Even if the user isjust walking, recognition results of all parts are notnecessarily walking. This section describes the method tointegrate them.
In this paper, we assume that if recognition result forhand is either posture, hand-posture or hand-gesture, itwill be a local activity, otherwise local activity will be null.
Then global activity is decided by a weighted majority
vote. We use recall of recognition result for training data
as a weight. Each body part has one voting right, but it isweighed, then the weighted votes are summed up over the
body except for local activity. For example, in Figure 2,
the recognition results of the left arm, hip, right leg, andleft leg are walk, walk, run, and walk, respectively.
Suppose recall of walk recognized with a sensor on the leftarm, hip, and left leg are 0.7, 0.8, 0.4, respectively, andrecall of run recognized with a sensor on right leg is 0.4,
poll to walk is 1.9 and poll to run is 0.4, resulting inwalking as global activity. Finally, combined activity is

WORKSHOP: HASCA

output by merging global and local activity. Even if a partof body is doing diﬀerent activity to global activity, user
activity is correctly recognized. For holding a mobile
phone while walking activity, the hand holding a phone isclassified into posture and recognized as holding a phone,whereas remaining parts would be classified into behavior
and recognized as walking.

training, the remaining 80% were used for testing. The
last one selectively uses SVM and DTW which learned
single activities only and integrate the results. Most partof the test data consists of global activity and gestures
sporadically occur. Correct recognition results for gestures
which is output in one second from the gesture ends areaccounted for true positives.

Evaluation

Results
Table 4 shows the recall and precision of recognition for
hand-gestures and hand-postures. “-” in the table meansthat the subjects were doing global activity only. The
recall and precision of gestures recognized by SVM werequite low. This is because the feature values haveinformation on the orientation and exercise intensity but
do not have information on the trajectory. Combined
gestures of behavior with hand gesture or hand posture
are almost misrecognized as single behavior. Because of
the same reason, the results for gestures while standing or
sitting are also low. In addition, sitting is not correctly
recognized. This is because the subjects leaned back inorder not to hit the armrest when performing a gesture,
resulting in the large diﬀerence to the training data forjust sitting. Also, gesture motion while sitting is slower
than that while standing, resulting in small value ofvariance, and average body orientation during the gestureis similar to just sitting. Moreover, all of hold mobile andraise hand while running were misrecognized as runningsince diﬀerence of hand orientation is absorbed by that ofvibration of running. From these results, it is hard forfeature-based recognition to recognize a lot of combined
activities.

In this section, we evaluate our system on the basis ofrecall and precision.
Setup
We evaluated our system for the data while standing,sitting, walking, running, and bicycling. The training andtest data were taken from three male subjects aged 22 to
28 years, who wore five accelerometers2 on their rightwrist, left wrist, hip, right ankle, and left ankle. The
sampling frequency was 20 [Hz]. Activities are as listed inTable 2: four postures (sitting, standing, lying, andkneeling), five behaviors (walking, running, bicycling,
descending stairs, and ascending stairs), five hand gestures(chop, throw, punch, draw a clockwise circle, and draw ananticlockwise circle), and two hand postures (holding amobile phone and raising a hand). The subjects wereacted these hand gestures and hand postures while sitting,standing, walking, running, and bicycling. Each gesture
was recorded 10 times for each global activity. All local
activities while ascending and descending, and jump andkick while bicycling were not performed for safety’s sake.
The logged data were manually labeled. In this evaluation,
activities were recognized with three method; SVM only,
DTW only, and our method. As shown in Table 3, thefirst two methods are comparisons, simply trained with all
the possible combinations. 20% of which were used for
2 Wirelss Technologies Inc.: http://www.wireless-t.jp/.

793

DTW, in the contrary, had high recall and precision for all
activities. It, however, is remarkable that the performanceof our proposal is comparable to that of DTW that has

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Table 2: Single activities

learned all the possible combined activities (9 global×7
local=63 activities), while our proposal has learned single
activities only (9 global+7 local=16 activities).
Table 4: Recall and precision of recognition.

Kind

Activity
SittingStanding
PostureLying
Kneeling
GlobalWalking
Running
Behavior Bicycling
Descending stairs
Ascending stairs
Chop
Throw
Hand
Punch
gestureLocal
Draw a clockwise circle
Draw a anticlockwise circle
Hand
Hold a mobile phone
posture Raise a hand

Table 3: Proposed andcomparison methods

Method# ofactivities
trained16

Proposal

SVM

63

DTW

63

Activities trained
4 postures
5 behaviors
5 hand gestures
while standing
2 hand postures
while standing
Combinations of 9
global ×
7 local activities
Combinations of
9 global ×
7 local activities

Activity
Global Local
Chop
Throw
Punch
Clockwise
Stand
Anticlockwise
Hold mobile
Raise hand
Null
Chop
Throw
Punch
Clockwise
Sit
Anticlockwise
Hold mobile
Raise hand
Null
Chop
Throw
Punch
Clockwise
Walk
Anticlockwise
Hold mobile
Raise hand
Null
Chop
Throw
Punch
Clockwise
Run
Anticlockwise
Hold mobile
Raise hand
Null
Chop
Throw
Punch
Clockwise
Bike
Anticlockwise
Hold mobile
Raise hand
Null
Average

Proposal
Recall Precision
1.000
0.917
1.000
1.000
1.000
1.000
1.000
1.000
1.000
0.882
0.996
1.000
0.999
1.000
1.000
0.955
1.000
1.000
1.000
1.000
1.000
1.000
0.955
1.000
1.000
1.000
1.000
1.000
1.000
1.000
0.929
0.833
0.944
0.944
1.000
0.944
0.705
0.442
0.988
0.917
0.826
1.000
0.975
0.429
0.429
0.500
0.500
0.000
0.000
0.000
0.000
0.981
0.900
1.000
0.913
0.917
1.000
0.742
0.622
1.000
0.500
0.450
1.000
0.999
0.991
0.843
0.857

SVM
Recall Precision
0.925
0.633
0.839
0.942
0.874
0.989
0.397
0.426
0.938
0.661
1.000
0.996
0.919
0.736
0.866
0.840
0.296
0.880
0.680
0.831
0.503
0.855
0.473
1.000
1.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.975
0.956
0.956
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.875
0.000
0.000
0.500
0.500
0.000
0.000
0.000
0.000
0.000
0.000
1.000
1.000
0.997
0.510
0.279

DTW
Recall Precision
1.000
0.950
0.996
1.000
0.964
0.996
1.000
1.000
1.000
0.920
1.000
1.000
0.993
0.993
0.908
0.842
0.983
0.920
0.625
0.805
0.914
1.000
0.471
1.000
1.000
0.522
0.994
1.000
0.946
1.000
0.983
1.000
0.993
0.995
0.990
0.990
1.000
0.999
0.537
0.986
1.000
0.894
0.997
0.966
1.000
0.978
0.916
0.975
0.942
0.886
0.734
0.309
0.729
0.956
0.872
0.843
0.930
0.711
0.780
0.908
0.937
0.686
1.000
1.000
0.878
0.908
0.908

794

The drawback of our proposal can be seen from theresults of hold mobile and raise hand while running. Theselow recall and precision were caused by the fact thatvibration of running is stronger than we assume, therefore
the hand is not classified into posture. Though our
method set the threshold according to the intensity of leg
as stated in the previous section, it is set based on walking
from our pilot study. Employing flexible threshold is our
future work. From our extra experiment, however, holding
mobile phone and raising hand are correctly recognized if
the hand is classified into posture.
Processing time
This section discusses the processing time of the proposed
method. Table 5 shows the processing time for the
displacement detection, constancy decision, andrecognition with SVM or DTW. Computer used for theevaluation is SONY VAIO VGN-US90PS (Inter CoreSolo
Processor 1.2 [GHz]). Simulation program is implemented
with Visual C++. The evaluation result is time per one
processing calculated based on the processing time for
100,000 trials. The reason the processing time for SVM
and DTW diﬀers in the proposal and comparisons is thatthe number of activities learned is diﬀerent. Most of therecognition with SVM is occupied with feature extraction,
therefore the eﬀect of the number of activities to berecognized is small. The recognition with SVM for the
proposal is smaller than that for the comparison method
because the proposal extracts only mean as feature value,
while the comparison method extracts mean and variance.
The number of template for DTW is one per one activity,
and the recognition with DTW takes processing time inresponse to the number of templates. For the results, theprocessing time for the displacement detection andconstancy decision is shorter than that for recognition andprocessing interval, therefore the proposed method can beWORKSHOP: HASCA

applied to real-time applications.

gesture, and global activity is behavior. Ex) Walking withpunch with the right hand and raising the left hand.

Table 5: Processing time for comparison methods andproposed method [msec].

Displacement detection
Constancy decision
Recognition with SVM
Recognition with DTW
Total

SVM

DTW

0.0531
0.0531

137
137

Posture0.00203
0.00344

Proposal
Behavior Gesture0.001410.04520.051434.9
0.098034.9

Recognition of complex activities
Local activities are confined to right-handed ones in thiswork. This section discusses recognition of gestures thatconsist of multiple local activities over the body. Even
when multiple parts of body perform local activities,
procedure of the proposed method does not change.
However, all gestures and postures for each part have to
be collected since gesture data for right hand cannot be
used for learning the same gesture for left hand.
Combination of local activities and global activities would
be the following six patterns: 1) Local activities are all
postures, and global activity is also posture. Ex) Standing
with holding cell phone with the right hand and raising
the left hand to hold on to a strap. 2) Local activities are
all postures, and global activity is behavior. Ex) Walking
with holding cell phone with the right hand and raising
the left hand to cross at the crosswalk. 3) Local activities
are all gestures and global activity is posture. Ex)
Standing with punch with the right hand and chop withthe left hand. 4) Local activities are all gestures andglobal activity is behavior. Ex) Walking with punch withthe right hand and chop with the left hand. 5) Local
activities are posture and gesture, and global activity is
posture. Ex) Standing with punch with the right hand andraising the left hand. 6) Local activities are posture and795

In the cases of 1) and 2), local activities are all postures,
which means no part of the body is performing gesture.
The parts of the body making a posture are classified into
posture and are recognized with SVM that has learned
postures only. The activity of the right hand is recognized
as standing with holding a mobile phone, left hand isrecognized as standing with raising the hand. The
remaining parts are recognized as standing or walking,therefore such combined activities can correctly berecognized. In the cases of 3) through 6), gesture isincluded as a local activity. Vibration of the gesture is
propagated to the opposite hand that is not performing
gesture. In this paper, since the part of body that
performs gesture has been limited to the right hand, thebody parts other than the right hand are not classified togesture. For example of walking with rotating the rightarm activity, vibration of the right hand is propagated to
the left hand and the left hand is classified into gesture.
In case that local activities are all gestures like in 3) and4), it is not problem that the left hand is classified togesture. Though the recognition may be incorrect due tothe propagated vibration, the system has a room torecognize it correctly. However, in the cases of 5) and 6),
recognition results of the hand making a posture when theleg performing gait are forced to be any of gestures. To
contend with the problem, recognition result whose
confidence is low is not output by setting threshold to the
DTW calculation. However, depending on the setting of
the threshold, correct recognition results may bediscarded.

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Conclusion
We constructed an activity recognition mechanism for
combined activities that classifies each part of body into
posture, behavior, and gesture, then recognizes individual
activities and integrates them. Evaluation results
confirmed that our proposed method achieved 0.84 ofrecall and 0.85 of precision, which is comparable to themethod that had learned all the combined activities: 0.90
of recall and 0.90 of precision. As a future work, we plan
to separate and integrate activities in more primitive level,that is to say to delete gesture component from raw dataof combined activity and to extract pure global activity.

References[1] Bao, L. and Intille, S.S.: Activity recognition fromuser-annotated acceleration data, in Pervasive 2004,
pp. 1–17 (2004).
[2] Pärkkäa, J., Ermes, M., Korpipää, P., Mäntyjärvi, J.,
Peltola, J., and Korhonen, I.: Activity classification
using realistic data from wearable sensors, in Trans. on
Information Technology in Biomedicine, Vol. 10, pp.
119–128, (2006).
[3] Murao, K. and Terada, T.: A Motion Recognition
Method by Constancy-Decision, in Proc. of ISWC
2010, pp. 69–72 (2010).
[4] Ravi, N., Dandekar, N., Mysore, P., and Littman,
M.L.: Activity Recognition from Accelerometer Data,in Proc. of AAAI 2005, pp. 1541–1546 (2005).
[5] Westeyn, T., Brashear, H., Atrash, A., and Starner, T.
Georgia tech gesture toolkit: Supporting experiments
in gesture recognition, in Proc. of ICMI 2003, pp.

796

85–92 (2003),[6] Junker, H., Amft, O., Lukowicz, P., and Tröster G.:
Gesture spotting with body-worn inertial sensors todetect user activities, in Journal of Pattern
Recognition Society, pp. 2010–2024 (2008).
[7] Keogh, E., Chu, S., Hart, D., and Pazzani, M.: An
online algorithm for segmenting time series, in Proc.
of ICDM 2001, pp. 289–296 (2001).
[8] Liu, J., Wang, Z., Zhong, L., Wiekramasuriya, J., andVasudevan, V. uwave: Accelerometer-based
personalized gesture recognition and its applications,in Proc. of PerCom 2009, pp. 1–9 (2009).
[9] Myers, C.S. and Rabiner, L.R. A comparative study ofseveral dynamic time-warping algorithms forconnected word recognition, in The Bell System
Technical Journal, vol. 60(7), pp. 1389–1409 (1981).
[10] Fujinami, K., Jin, C., and Kouchi, S.: Tracking
On-body Location of a Mobile Phone, in Proc. of
ISWC 2010, Late Breaking Results, pp. 190–197(2010).
[11] Junker, H., Lukowicz, P., and Tröster, G.: Sampling
Frequency, Signal Resolution and the Accuracy of
Wearable Context Recognition Systems, in Proc. of
ISWC 2004, pp. 176–177 (2004).
[12] Intille S.S., Bao, L., Tapia, E.M. and Rondoni, J.:
Acquiring in Situ Training Data for Context-aware
Ubiquitous Computing Application, in Proc. of CHI
2004, pp. 1–9 (2004).
[13] Vapnik, V.: The Nature of Statistical Learning
Theory, Springer (1995).