Recognizing the Use of Portable Electrical
Devices with Hand-Worn Magnetic Sensors
Takuya Maekawa, Yasue Kishino, Yasushi Sakurai, and Takayuki Suyama
NTT Communication Science Laboratories
2-4 Hikaridai Seika-cho, Souraku-gun, Kyoto, Japan{surname.name}@lab.ntt.co.jp

Abstract. The new method proposed here recognizes the use of portableelectrical devices such as digital cameras, cellphones, electric shavers,
and video game players with hand-worn magnetic sensors by sensing themagnetic fields emitted by these devices. Because we live surrounded bylarge numbers of electrical devices and frequently use these devices, we
can estimate high-level daily activities by recognizing the use of electrical devices. Therefore, many studies have attempted to recognize theuse of electrical devices with such approaches as ubiquitous sensing andinfrastructure-mediated sensing. A feature of our method is that we canrecognize the use of electrical devices that are not connected to the homeinfrastructure without the need for any ubiquitous sensors attached tothe devices. We evaluated the performance of our recognition method
in real home environments, and confirmed that we could achieve highly
accurate recognition with small numbers of hand-worn magnetic sensors.
Keywords: Wearable sensing; Activity recognition; Magnetic sensor.

1

Introduction

Activity recognition is one of the most important technologies in relation tocontext-aware and lifelogging applications, e.g., elder care support and ﬁtness
monitoring. We can categorize activity recognition technologies into two main approaches; wearable sensing and environment augmentation. The wearable sensing
approach recognizes activities by using sensor data obtained from such bodyworn sensors as accelerometers and microphones [1,11,12,13]. In many cases, theenvironment augmentation approach uses ubiquitous sensors such as RFID tags
and/or switch sensors installed in the environment [20,17,19,14]. Although theenvironment augmentation approach places a smaller burden on the user than
the wearable sensor approach, ubiquitous sensors are expensive to deploy because
we have to attach them to various indoor objects and maintain a large numberof them. On the other hand, several studies have used a sensor device attached
to a single point in the environment [5,16]. For example, [16] recognizes electrical device use by monitoring electrical noise on residential power lines. Many
environment augmentation approaches detect the use of daily objects in the environment by using small sensors attached to the objects and/or a single point
K. Lyons, J. Hightower, and E.M. Huang (Eds.): Pervasive 2011, LNCS 6696, pp. 276–293, 2011.
c Springer-Verlag Berlin Heidelberg 2011


Recognizing Use of Portable Electrical Devices

277

device that monitors the objects and then estimates high-level daily activities
using the detected information. This approach is based on the idea that objects
in use relate to an activity the user is performing. For example, when a user isusing a razor and preshave lotion, we can easily assume that the user is shaving.
On the other hand, because many wearable sensing approaches use bodyworn accelerometers, they can recognize only simple low-level activities such as
walking and running. However, unlike the environment augmentation approach,
the wearable sensing approach can sense users’ activities in both indoor andoutdoor environments. Also, as mentioned above, its deployment cost is smaller
than that of the environment augmentation approach. In this paper, we try torecognize the use of daily objects solely by employing wearable sensors that havethe above advantages. That is, we recognize which object is in use by employing
only wearable sensors without any sensors embedded in the user’s environment.
This permits us to realize a low deployment cost and place-independent highlevel activity recognition. In this work, by using hand-worn magnetic sensors,
we try to recognize the use of portable electrical devices such as digital cameras, cellphones, electric shavers, video game players, and music players. Thatis, we sense the magnetic ﬁelds that the devices emit and determine the device a
wearer is using by analyzing the sensor data. In modern societies, because we are
surrounded by large numbers of electrical devices, we can estimate high-level activities by recognizing the use of these devices. For example, when we recognizethat a wearer is using a hair dryer, we can know that she is doing her hair. Suchhigh-level activity recognition is a fundamental technology of context-aware systems and lifelogging. Therefore, many studies have been undertaken with theaim of recognizing the use of electrical devices. However, several approaches require a sensor node for each electrical device [8]. Also, because other approaches
utilize the existing infrastructure in a home, i.e., power lines [16], they cannot
recognize the use of electrical devices that are not connected to the infrastructure
via power outlets. Because the new method proposed in this paper recognizes
the use of portable electrical devices by employing a magnetic ﬁeld emitted frommagnets and ICs embedded in the devices, we can recognize the use of devices
that are not connected to the infrastructure. In addition, we can recognize theuse of the devices in outdoor environments. Portable electrical equipment suchas digital cameras and cellphones are frequently used out of doors.
In the rest of this paper, we ﬁrst introduce work related to activity recognition
with sensor data, and then describe the magnetic ﬁeld emitted from electricaldevices and the mechanism of the magnetic sensors. After that, we describe
the design and implementation of our proposed sensor device, and explain our
method, which recognizes the use of portable electrical devices by employing
magnetic sensor data. By using the sensor device, we collect sensor data in threeactual home environments and then evaluate our method by using the data. The
contributions of this paper are that we propose a new method that recognizes
the use of portable electrical devices without the need to install any sensors inenvironments, i.e., without attaching sensor nodes to the devices. In addition,
we experimentally investigate the appropriate number and locations of magnetic

278

T. Maekawa et al.

sensors attached to wearer’s hands. To achieve this, we attempt to recognize theuse of devices accurately with small numbers of sensors. This enables us to reducethe burden it places on wearers.

2

Related WorkAs mentioned in section 1, activity recognition methods are categorized into environment augmentation and wearable sensing approaches. Many environment
augmentation approaches employ a large number of small sensors such as switch
sensors, RFID tags, and accelerometers installed in the corresponding environments [20,17,19]. Although the approach can achieve ﬁne-grained measurementsof daily lives, its deployment and maintenance costs, e.g., costs related to battery
replacement, are very large. Several studies employ a single point sensor device
that can monitor home infrastructures to detect the use of electricity, water, or
gas in home environments [3,5,16]. However, the approach cannot monitor theuse of devices and objects that are not connected to the home infrastructuresuch as the plumbing or electrical systems. The method proposed in this paper
can recognize the use of portable electrical devices that are not connected tothe home infrastructure by employing wearable sensors. However, as described
below, our approach may not be able to recognize the use of many large stationary electrical devices such as washing machines and refrigerators, which areusually connected to power lines. We consider that, by combining our methodwith the infrastructure-based method, we can recognize the use of both portable
and stationary electrical devices in both indoor and outdoor environments withsmall deployment costs.
Most wearable sensing approaches use multiple accelerometers attached to
the wearer’s body [1,18]. Although these approaches can recognize the wearer’s
activities in outdoor environments, they place a burden on the wearer because
she has to wear several sensors. Also, unlike approaches that leverage the useof daily objects, the wearable sensing approaches recognize just simple low-level
activities such as walking and running because they only use accelerometers. On
the other hand, some studies can recognize high-level activities by employing
such rich sensors as body-worn microphones and cameras [2,10,12,13,15]. However, the methods that employ such rich sensors as cameras and microphones
may generate privacy concerns. Because the method proposed in this paper employs magnetic sensors, there is less of a privacy issue than with camera and/or
microphone based methods. The magnetic sensor simply outputs a sequence ofnumerical values in the same way as accelerometers. In addition, rich sensors
such as cameras and microphones must handle a large amount of data and/or
consume a lot of energy.

3Magnetic Field and Magnetic Sensor

Before explaining our approach, we describe the magnetic ﬁeld emitted by electrical devices and the magnetic sensor mechanism.

Recognizing Use of Portable Electrical Devices

3.1

279

Magnetic Field Emitted by Electrical Devices

The magnetic ﬁelds in electrical devices have two main sources. The ﬁrst sourceis permanent magnets embedded in the devices. Because permanent magnets arewidely used components of motors, speakers, and earphones, they are familiarin our daily lives. The intensity of the magnetic ﬁeld emitted by a permanent
magnet attenuates greatly according to the distance from the magnet. The attenuation feature depends on the form and strength of the magnet. With ahorizontally-located pillar-shaped magnet 5 mm in diameter, 3 mm thick, andwith a residual magnetization of 1.2 T, for example, the magnetic ﬂux density
values 5 mm and 10 mm away in the vertical direction are 36.0 and 7.1 mT,
respectively. 1.2 T magnets are commonly used as speaker components. The second source is the ﬂow of an electric current in the devices. An electric current
ﬂowing through a device produces a magnetic ﬁeld, and the intensity of themagnetic ﬁeld also attenuates according to the distance. Such a magnetic ﬁeld
is emitted by conductive wires, coils, and ICs included in electrical devices. On
the other hand, to restrict the harmful eﬀects of a time-varying magnetic ﬁeld
on a person’s health, some organizations have established exposure guidelines[4,6]. Based on these guidelines, many electrical devices are designed to limit
magnetic ﬁeld leakage by surrounding the magnetic sources with such magnetic
materials as iron. This eﬀect is called the shielding eﬀect. The intensity of magnetic ﬁelds emitted by conductive wires and ICs is much smaller than that bypermanent magnets and coils, and the intensity is also smaller than that of theearth’s magnetism (0.03 - 0.04 mT) in many cases. It is diﬃcult to capture thesmall magnetic ﬁelds by using body-worn magnetic sensors that are aﬀected by
the earth’s magnetism. Therefore, we mainly focus on permanent magnets andcoils.
3.2

Magnetic Sensor

Magnetic ﬁeld detection has various applications such as current sensing, orientation sensing, diagnosis of disease, and paper currency validation, and sovarious kinds of magnetic sensors have been developed [9]. Here, we introduce
the widely used Hall eﬀect sensors. A Hall eﬀect sensor is a kind of magnetic
sensor that utilizes the electromotive force integrated in an electrical conductor
carrying a current when it is placed in a magnetic ﬁeld perpendicular to thecurrent. A Hall eﬀect sensor outputs a voltage in proportion to the magnetic
ﬂux density that penetrates its element. Because of its simple mechanism, verysmall and inexpensive sensors, e.g., from about 1 to 2 mm square, have becomecommercially available.

4

Sensor Device Design

4.1

Manual Use of Electrical Devices

To obtain a design guideline for our sensor device, we ﬁrst categorize the waysof using electrical devices into two types. The ﬁrst is that the user picks up the

280

T. Maekawa et al.

device and uses it. The second is where the user employs a ﬁxed static device bypushing its buttons, turning its knobs, etc. The ways of using many large devices
such as refrigerators and washing machines fall into the second category. Most
large devices are connected to power outlets, and so the methods that monitor
power line infrastructures can recognize the use of these devices [16]. On theother hand, most of the devices employed in the ﬁrst way are small because theuser holds the devices in her hand when using them. Also, because these devices
are portable, many of them are battery powered, which makes them impossible to recognize with the methods that monitor power line infrastructures. As
mentioned in the previous section, electrical devices emit magnetic ﬁelds fromtheir embedded magnets, coils, etc. The idea we propose in this paper is that we
recognize the electrical device a user is using by employing the magnetic ﬁeld.
In particular, we recognize the use of portable electrical devices with hand-worn
magnetic sensors. As described in the previous section, a magnetic ﬁeld attenuates greatly because of its characteristics and the shielding eﬀect. Thus, if auser holding a portable device wears a sensor on her hand, we can eﬀectively
detect the magnetic ﬁeld emitted by the device. Then, we analyze the detectedmagnetic ﬁeld and recognize which device the wearer is using. We detail therecognition method later.
Here, we examine the parts of the hands to which we should attach magnetic
sensors. We consider there to be two criteria for selecting the parts. (i) Although
the magnetic sensors are small, we should select locations where the attachedsensors do not place a large burden on the wearer. For example, when we attachsensors to a person’s ﬁngertips, the sensors may impede the operation of buttons
and keyboards. Moreover, if we attach sensors to the regions surrounding theﬁnger joints, they will be uncomfortable when the wearer bends her ﬁngers. (ii)
We should attach sensors to locations where the sensors can eﬀectively detect
the magnetic ﬁelds emitted by portable devices held in the wearer’s hands. As
mentioned in the previous section, because magnetic ﬁelds attenuate greatly, we
should attach sensors to locations where they can monitor devices near them.
Here, we look at the ways of holding various kinds of portable electrical devices
in the hands. We found that button position and shape of a device aﬀect the way
it is held as shown in Fig. 1. For example, when holding a device with buttons asshown in Fig. 1 (a) and (c), we usually hold the device in a way that enables us toaccess the buttons with our ﬁngers. Moreover, a device with a handle, as shown in
Fig. 1 (c), is normally held by the handle. As above, we basically hold the devices
with the ﬁngers and palm. In addition to the shape and button position, its sizeaﬀects the way it is held. We often hold small and lightweight devices with just
our ﬁngers. For example, when holding a small camera as shown in Fig. 1 (a),we can hold it using only our ﬁngers. We hold slightly larger and heavier devices
with part of the palm in addition to the ﬁngers, e.g., when holding a relatively
large smart phone as shown in Fig. 1 (b). That is, we hold devices mainly withthe ﬁngers and also with part of the palm. Based on the above facts, we decided
to sense the magnetic ﬁelds from portable devices by mainly focusing on theﬁngers. In particular, we focus on the parts of the ﬁngers where sensors would

Recognizing Use of Portable Electrical Devices

(a)

(b)

281

(c)

Fig. 1. Ways of handling portable devices: (a) digital camera, (b) smart phone, and(c) hand-held cleaner. Regions circled with dashed lines represent those parts of thedevices that the hand touches.

not impose a large burden on the wearer, i.e., between the ﬁrst and second joints
and between the second joint and the base of the ﬁnger. We considered that,by embedding a magnetic sensor in a ﬁnger ring, which is usually worn between
ﬁnger joints, we could achieve a low burden and convenient sensing.
In our investigation, we also found that the way a portable device is held
depends on the device (or the operation of the device). As mentioned above, theways devices are held are determined by their shape and the positions of theirbuttons. That is, the way a device is held is almost identical whenever the user
performs a certain operation.
4.2

Basic Idea of Recognition Method

As described above, we try to distinguish which portable electrical device a user
is using by analyzing time-series sensor data obtained from hand-worn magnetic
sensors that sense magnetic ﬁelds emitted by various components embeddedin the devices. Therefore, we should solve the multi-class recognition problem,
namely we must classify sensor data at each time slice into an appropriate activity class (use of an electrical device). Here, to solve the problem, we look at two
discriminative characteristics of sensor data obtained from hand-worn magnetic
sensors. The ﬁrst is strength of magnetic ﬁeld (magnetic ﬂux distribution). Assume that a permanent magnet is embedded at a certain position on a device.
When the way of holding the device is identical whenever the wearer performs
a certain operation, a magnetic sensor ﬁxed to a certain hand position may output similar sensor values whenever the operation is performed. This is because
the strength of a magnetic ﬁeld depends on the intensity of the magnet anddistance from the magnet. Because the intensity and positions of the embedded
magnets diﬀer from device to device, we consider that we can recognize which
device a wearer is using by utilizing the strength of a magnetic ﬁeld picked upby hand-worn magnetic sensors. The second discriminative feature is the temporal change in the strength of a magnetic ﬁeld. For example, when the amountof current ﬂowing in a device changes or a wearer changes the way of holding
a device when operating it, the strength of the magnetic ﬁeld obtained from ahand-worn magnetic sensor may change. Such characteristic temporal variationsin the magnetic ﬁeld can allow us to distinguish between activity classes.

282

T. Maekawa et al.

R-T1
1.5

R-I1 R-I2
R-M2
R-M1
R-P1
R-R1 R-R2

Sensor output [V]

1.4
1.3
1.2
1.1
1
0.9

R-L1 R-L2

Example magnetic sensor data
0.8
Time

Fig. 2. A magnetic sensor shown next to a quar- Fig. 3. Time-series raw magter dollar coin (left) and our prototype sensor glove netic sensor data of R-I1 ob(right). We provided each sensor on the glove with an tained when we brought a cellidentifier. The identifier is labeled with an initial let- phone close to the sensor andter indicating the left or right hand, an initial letter then moved the phone away
showing its position, and a serial number. For exam- from the sensor. (The voltage
ple, the identifier of the first sensor attached to the values are shifted.)
little finger of the right hand is R-L1.

4.3

Design and Implementation of Prototype Device

Based on the characteristics of a magnetic ﬁeld mentioned in section 3.1 andthe ways of using portable electrical devices mentioned in section 4.1, we design
and implement a sensor device that focuses on the ﬁngers and palm. However,
it is not clear which of the ﬁnger parts that we focus on are actually eﬀective
for recognizing the use of portable devices. Moreover, it is unclear how many
sensors are needed to achieve highly accurate recognition. Thus, in this paper,
we ﬁrst develop a sensor device that is equipped with many sensors at important
locations on the hand mentioned above, and then we determine the good locations experimentally. The prototype device developed in this study is in the formof a glove with ten magnetic sensors as shown in Fig. 2. We attached magnetic
sensors to the important parts of the hand mentioned in section 4.1. We also
attached a magnetic sensor to the center of the palm. We used Asahi Kasei’s
HW105A Hall ICs as magnetic sensors, and also used a glove that ﬁtted the handwell. As shown in Fig. 2, the sensor is suﬃciently small. Because the device isa prototype, these sensors are connected to a USB port of a host PC via cables
and a sensor board. The board samples the sensor data at a sampling rate ofabout 330 Hz. We implemented the devices for both right and left hands. If we
can recognize the use of electrical devices with a small number of sensors, we
can achieve convenient sensing by using a small ﬁnger ring shaped device witha magnetic sensor because we assume that the sensors will be attached between
ﬁnger joints. Note that several magnetic sensor products are sensitive to temperature changes. That is, output voltage characteristics of the sensors change
according to ambient temperature. When we use such sensors, we should correct
sensor outputs by using temperature data.

Recognizing Use of Portable Electrical Devices

pick up camera

283

release camera

hold camera(a) `Take picture’(b) `Take picture’ 1 day later(c) `Take picture’ 12 days later (d) `Take picture’ 27 days later(e) `Take picture’ close to tower (f) `Take picture’ close to railway

characteristic frequency

synchronizing(g) `Use hair dryer’(h) `Take video’Fig. 4. Sensor data sequences obtained from our devices. Positive or negative sensor
data values indicate the direction of the magnetic flux that penetrate a sensor.

4.4

Characteristics of Sensor Data

We examine sensor data obtained from our implemented sensor device. Fig. 4(a)-(f) show time-series sensor data sequences obtained from sensors attachedto the middle ﬁngers of both hands when the wearer was performing a certain
activity at diﬀerent times and at diﬀerent locations. (The sensor identiﬁers are LM1 and R-M1 as shown in Fig. 2.) The activity consists of the wearer picking up
a digital camera, turning on the camera, taking a picture, turning oﬀ the camera,
and then releasing the camera. The x- and y-axes of the graphs represent time
and the output voltage of the magnetic sensor, respectively. (The voltage values
are ampliﬁed, smoothed, and then shifted.) From Fig. 4 (a)-(d), we can ﬁnd thatthe sensor data values obtained on diﬀerent days while holding the camera weresimilar. Also, Fig. 4 (e) and (f), respectively, show sensor data sequences obtained
at places close to a power transmission tower and an electric train line, which may
generate magnetic ﬁelds. As mentioned in section 3.1, because the magnetic ﬁeld
attenuates greatly, magnetic sensors were not aﬀected by magnetic ﬁeld sources
located at a distance. In addition, although these data change depending on theorientation of the sensor as a result of the eﬀect of the earth’s magnetism, theamount of change was small. On the other hand, Fig. 4 (g) shows sequences of
sensor data obtained when the wearer was using a hair dryer. The sequences are
diﬀerent from those obtained when the wearer was taking a picture.

5

Recognition Method

We attempted to recognize the use of electrical devices by using the sensor data
sequences introduced above. Fig. 5 shows the architecture of our recognition
method. In this architecture, we ﬁrst amplify the output voltages from handworn magnetic sensors and then denoise them. As shown in Fig. 3, we can detect
the presence of high-frequency noise in a sequence of raw output voltage signals.
Therefore, we remove the noise by employing a low-pass ﬁlter (moving average).

284

T. Maekawa et al.

Board
Amplifier

Generative Classifiers

Decision tree 1

HMM 1

Decision tree 2

HMM 2
…

…

…

Decision tree n…

…

Sensor glove

Low-pass
FilterFeatureExtraction

Discriminative Classifiers…

Sensordata

Host PC

HMM
Likelihoods

HMM nFig. 5. Architecture of our recognition method

The charts in Fig. 4 show smoothed signals. The host PC samples the smoothed
sensor data from the board at a sampling rate of 33 Hz. The host PC thenextracts features from the data and classiﬁes the extracted feature vectors at eachtime slice into the appropriate activity class. We describe the feature extraction
and the classiﬁcation method below.
5.1

Feature Extraction

We extract features from sensor data that are used to model/recognize activity
classes. We compute features from each sample of sensor data. Thus, because
the host PC samples sensor data from the board at a sampling rate of 33 Hz,
about 30 feature vectors are generated per second and become inputs of theclassiﬁcation method. In addition to the smoothed output voltage values fromthe hand-worn magnetic sensors, we also use (i) energy, (ii) dominant frequency,and (iii) the diﬀerence between the output voltage values of diﬀerent sensors asfeatures.
The energy and dominant frequency are computed for each sensor data sequence. We extract the energy and dominant frequency features based on the
FFT components of each 64-sample window because we can ﬁnd a characteristic frequency in the sensor data captured during the performance of several
activities as shown in Fig. 4 (g), which may be caused by motor rotation. Theenergy can be used to distinguish low intensity changes in sensor data from highintensity changes. The energy feature is calculated by summing the magnitudes
of squared discrete FFT components. For normalization, the sum was divided
by the window length. Note that the DC component of the FFT is excluded
from this summation. The dominant frequency is the frequency that has thelargest FFT component, and this component should be ten times larger than
the average component of all the frequencies in this implementation.
We also compute the diﬀerence between the output voltage values of two
diﬀerent sensors sampled at the same time. We compute the diﬀerences for all
combinations of two sensors attached to the same hand, and then use them asfeatures. As mentioned in section 4.4, magnetic sensors are aﬀected by the earth’s
magnetism. Fig. 4 (h) shows sequences of sensor data obtained during digital
camcorder use. We found temporal changes in the sensor data sequences and thechanges of these sequences were synchronized. This was caused by changes in
the wearer’s orientation when using the camcorder. That is, the magnetic sensor

Recognizing Use of Portable Electrical Devices

285

data change according to the orientation of the sensor. Our idea is that, by using
the diﬀerence between the sensor data values of two sensors, we cancel out thechanges caused by earth’s magnetism (the orientation change).
As described above, we use the smoothed sensor data value, energy, dominantfrequency, and diﬀerence as features. When a user wears 10 sensors on the right handand 10 sensors on the left hand, the generated feature vector consists of a total of
150 feature values ((10 + 10 + 10 + 45) × 2), i.e., the vector has 150 dimensions.
5.2

Classification Methodology

We classify an extracted feature vector in an appropriate class by employing
supervised machine learning techniques. That is, we ﬁrst model each activity(use of a device) by using labeled training data and then recognize test data withthe learned models. Note that a label includes information about the class labelof its related activity and activity start and end times. Here, the classiﬁcation
approaches used in machine learning are divided into two groups: one groupuses discriminative techniques that learn the class boundaries, and the otheruses generative techniques that model the conditional density functions of thedata classes. The classiﬁcation performance of the discriminative techniques,
which ﬁnd the discriminant boundaries of the classes, often outperform those ofgenerative techniques. By contrast, handling missing data is often easier withthe generative techniques. State of the art activity recognition studies achieve
high accuracy by employing a hybrid discriminative/generative approach thatcan combine the advantages of the two techniques [7,10,11,13].
These facts provide our motivation for using the hybrid discriminative/
generative approach shown in Fig. 5. Our classiﬁcation method employs two
main modules: discriminative classiﬁers and generative classiﬁers. The input of
the ﬁrst module is the extracted feature vector sequence. The ﬁrst module consists of some decision tree based binary classiﬁers trained with feature vectors.
We build each decision tree to recognize its corresponding activity class. Thatis, the number of decision trees n corresponds to the number of activities themethod learns. Each decision tree computes its associated class probability foreach feature vector in the feature vector sequence. That is, each decision tree
outputs the class probability sequence. For example, a decision tree for the ‘vacuum activity’ class outputs the probability of the class for each feature vector.

Table 1. Information about experimental
environments and participantsHousesType#rooms#days#sessions#residents
Age
GenderHouse A House B House C
house
7
8
10
3
60
male

house
5
4
6
3
35
male

apartment
2
7
12
1
32
male

Table 2. Activities performed in our
experiment
Activities
A talk on cellphone
B operate cellphone
C use smart phone
D listen to music
E shave
F use hair dryer
G brush teeth

Activities
H vacuum
I watch TV
J play video gameK listen to radio
L use flashlight
M take picture
N take video

286

T. Maekawa et al.

The input of the second module consists of n-dimensional class probability sequences computed by the n decision trees. The second module also comprises n
HMM classiﬁers [21], which can be used to recognize signals with temporal patterns, trained with a sequence of output class probabilities of the discriminativeclassiﬁers. We also build each HMM to recognize its corresponding activity class,that is, each HMM also outputs the likelihood of its corresponding activity. The
class with the highest likelihood is the classiﬁed class.

6

Evaluation

In this section, we evaluate our activity recognition approach by using sensor
data obtained in real environments. We also investigate how many magnetic
sensors are required and to which parts of the hands we should attach them toachieve highly accurate activity recognition that imposes the minimum burden
on the wearer.
6.1

Data Set

We collected sensor data in three diﬀerent dwellings; one seven-room house(house A), one ﬁve-room house (house B), and one two-room apartment (house
C). One resident (an experimental participant) in each house wore our prototype
devices on both hands and collected sensor data. Our devices were connected
to a laptop PC in a backpack via cables. To annotate the collected sensor data,
each participant also wore a head-mounted camera that captured the region in
front of the participant’s body. Table 1 shows an overview of the experimental
conditions. During the experimental periods, the participants collected sensor
data without being supervised by researchers.
Here, the most natural data would be acquired from the normal daily livesof the participants. However, obtaining suﬃcient samples of such data is very
costly. We collect sensor data by using a semi-naturalistic collection protocol [1]that permits greater variability in participant behavior than laboratory data.
In the protocol, participants perform a random sequence of activities (obstacles) following instructions on a worksheet. The participants are relatively free
as regards how they perform each activity because the instructions on the worksheet are not very strict, e.g., “shave your face” and “vacuum the room with ahand-held cleaner.” During the experimental period, the participants completed
data collection sessions that included the random sequence of activities (use of
portable electrical devices) listed in Table 2. We selected these portable devices(activities) from those frequently found in appliance stores and online stores.
Here we describe how these activities were performed in detail. In activity
B, each participant operated his cellphone, i.e., texting and dialing. In activity
C, we instructed the participants to browse an arbitrary web page on a smart
phone. In activity E, each participant shaved his face with an electric shaver.
In activity G, each participant brushed his teeth with an electric toothbrush. In
activity H, we instructed the participants to vacuum their house with a handheld cleaner. In activity I, we instructed the participants to operate a TV with aRecognizing Use of Portable Electrical Devices

287

remote control, e.g., control its volume and switch channels. In activity J, eachparticipant operated a video-game console such as Nintendo DS or Wii in his
house. In activity K, we instructed the participants to listen to a radio programon a portable radio. In activity M, we instructed the participants to take apicture with a digital camera. In activity N, we instructed the participants totake a video with a (digital) camcorder. Here, the electrical devices used in theexperiment were located in their appropriate places in each house. For example,
an electric toothbrush was placed on a wash stand. Note that, with respect todevices that are usually used in various places such as cellphones and digital
cameras, we instructed the participants to use the devices in various places bothin and outside the house. Also, when a participant did not have a device, we
asked him to buy the device. (We paid for it.)
6.2

Evaluation Results

We evaluated the performance of our approach by using the collected and annotated sensor data. We conducted a ‘leave-one-session-out’ cross validation evaluation. That is, we tested one session obtained in a house by using classiﬁers
trained on other sessions obtained in the same house. To evaluate the performance of our method, we used the precision and recall calculated based on the
results for the estimated class at each time slice.
Performance of our method. The left part of Table 3 shows the precision andrecall of our recognition method in each house when we used sensor data obtained
from all 20 magnetic sensors. As shown in the table, our method achieved very
high recognition accuracy simply by using wearable sensors. We achieved about
80% average precision and recall in each house. The accuracies for houses A and C
were higher than that for house B. This may be because the amounts of training
data in houses A and C were suﬃcient. With large amounts of training data, we
can capture various ways of using electrical devices. However, in house A, theprecision of the ‘take picture’ activity was not good. As shown in the confusion
matrix of house A in Table 4, other activities such as ‘use smart phone’ and‘watch TV’ were mistakenly classiﬁed in the ‘take picture’ class. This may be
because we could not model the ‘take picture’ class well. When the participant
in house A took a picture, he changed the way he held the camera slightly
depending on the type of photograph he was taking, e.g., closeup photograph or
telephotograph. Although the changes were very small, the data values obtained
from the sensors were very diﬀerent as shown in Fig. 6 (a) and (b). The timeseries sensor data sequences in Fig. 6 (a) were obtained when the participant inhouse A took a picture of the landscape in front of him. The time-series sensor
data sequences in Fig. 6 (b) were obtained when the participant took a pictureof an object by pointing the camera directly at it. Although the data values
obtained with the R-M2 sensor were similar to each other, those obtained with
the L-M1 sensor were diﬀerent. This was caused by the magnetic ﬁeld, which
attenuates greatly with distance, as mentioned in section 3.1. Depending on thesensor position, the sensor data values change considerably. This phenomenon

288

T. Maekawa et al.

Table 3. Accuracies (precision / recall) of the recognition method in each house. The
values are percentages. The left portion of this table shows the accuracies when we use
the sensor data from all 20 sensors. The right portion shows the accuracies when we
use the sensor data of only the top-4 contributing sensors.
All 20 sensors
Only top-4 sensors
House A House B House C House A House B House C
A: talk on cellphone 95.1/82.2

25.2/80.2 27.2/89.9 89.6/88.8 58.5/43.1 43.7/75.0
89.4/73.3 98.2/80.4 70.0/79.9 58.1/73.5 91.9/89.7
92.5/76.0 91.4/85.5 91.6/83.6 68.0/64.9 77.4/61.7 98.4/93.4
87.2/71.0 63.4/65.5 73.0/89.5 75.1/72.8 59.6/37.2 81.3/89.5
78.8/78.8 99.9/100.0 93.9/68.8 84.9/84.9 86.4/71.8 81.1/67.1
95.9/99.4 94.4/89.9 91.2/89.3 91.4/93.7 96.1/95.5 97.2/94.0
95.4/95.5 79.6/51.2 79.2/67.4 65.7/79.0 50.8/81.0 55.7/72.0
90.0/83.3 99.4/66.0 98.0/85.4 82.8/79.6 26.6/58.5 93.6/86.2
79.4/78.9 84.3/73.8 89.3/69.4 76.8/74.1 72.3/69.5 95.2/83.5
95.5/97.7 99.3/99.5 95.9/85.8 88.8/96.0 99.1/100.0 88.8/89.8
89.0/73.6 92.9/81.2 69.8/70.1 59.3/41.5 74.9/78.9 72.7/73.2
86.9/90.7 95.2/80.0 98.3/71.6 58.3/80.8 22.7/21.6 80.9/73.2
35.3/77.2 98.4/92.0 90.8/99.9 82.5/82.8 99.0/88.8 73.8/75.2
91.4/78.7 53.3/28.7 99.3/92.2 96.7/82.2 40.6/29.0 98.0/80.9

B: operate cellphone 89.5/84.9
C: use smart phone
D: listen to music
E: shave
F: use hair dryer
G: brush teeth
H: vacuum
I: watch TV
J: play video game
K: listen to radio
L: use flashlight
M: take picture
N: take video
Average
Overall

85.9/83.4 83.3/76.2 85.4/81.7 77.9/78.7 65.9/65.0 82.3/81.6
83.3/83.3 79.6/79.6 81.2/81.2 77.6/77.6 68.8/68.8 82.7/82.7

was also observed for ‘talk on cellphone’ in houses B and C. However, this result
indicates that, when we have suﬃcient amounts of training data, we may beable to achieve very ﬁne-grained recognition of electrical device operation, e.g.,
distinguish a closeup photograph from a telephotograph. As above, the various
ways of holding the electrical devices reduced the accuracy in our experiment.
We found another reason for recognition failure, namely the positions of thecomponents in electrical devices. The accuracies of ‘brush teeth’ and ‘take video’
in house B were poor because the intensities of magnetic ﬁelds that the handworn sensors sensed in these activities were very small. That is, the magnetic
components included in the devices that were used in these activities may havebeen located far from these sensors when the devices were used. It is very diﬃcult
to recognize the use of such devices with our approach because our approach
employs the intensity of the magnetic ﬁeld. This problem occurred with ‘brush
teeth’ and ‘take video’ in house B, and ‘watch TV’ in house C. However, although
the hand-worn sensors could not sense the high intensity magnetic ﬁeld emitted
by the electric toothbrush in house B, the accuracy of ‘brush teeth’ in house
B was not very bad. When the participant in house B brushed his teeth, hecontrolled the toothbrush so that it brushed his front teeth, back teeth, upper
teeth, and lower teeth. That is, he moved his hand to control the toothbrush andthen the posture of his hand changed. In section 4.4, we mentioned that magnetic
sensors are slightly aﬀected by the earth’s magnetism. When the posture of
the hand changes, sensor data obtained from hand-worn sensors also change

Recognizing Use of Portable Electrical Devices289

N: take video

M: take picture

L: use flashlight

K: listen to radio

J: play video game

I: watch TV

H: vacuum

G: brush teeth

F: use hair dryer

E: shave

D: listen to music

C: use smart phone

B: operate cellphone

A: talk on cellphone

Table 4. Confusion matrix in house A when we use sensor data of all 20 sensors. The
values are percentages.

A 82.2 0.0 2.2 0.0 0.0 0.0 4.7 0.0 9.1 0.0 0.0 0.9 0.8 0.0
B 0.0 84.9 0.0 0.7 0.4 0.0 0.2 0.1 0.2 0.0 0.4 2.4 9.4 1.3
C 0.0 0.7 76.0 0.0 0.3 0.0 0.0 0.0 3.7 5.6 0.0 1.1 12.2 0.4
D 0.5 4.2 0.6 71.0 5.1 0.3 0.6 0.6 1.1 0.8 6.0 0.6 6.4 2.0
E 0.2 3.0 0.0 0.9 78.8 0.3 0.1 2.2 0.9 0.0 0.4 0.0 12.5 0.6
F 0.3 0.0 0.0 0.0 0.0 99.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3
G 0.0 0.0 0.0 0.7 0.0 0.0 95.5 0.0 1.2 0.0 0.0 0.0 0.4 2.2
H 0.0 0.8 0.0 0.1 0.0 0.0 0.0 83.3 0.0 2.2 0.1 2.6 10.6 0.3
I 0.0 0.0 2.0 0.0 0.2 0.5 0.0 0.5 78.9 0.1 1.7 0.0 15.3 0.8
J 0.0 0.2 0.1 1.1 0.1 0.0 0.0 0.1 0.2 97.7 0.0 0.0 0.5 0.0
K 0.0 2.0 1.3 4.2 2.2 0.0 0.8 1.5 4.8 0.4 73.6 0.0 9.1 0.0
L 0.0 0.2 0.0 0.0 0.0 0.0 0.0 1.8 0.8 0.2 0.0 90.7 6.3 0.0
M 0.0 0.0 0.4 2.0 10.2 0.8 0.4 0.0 3.9 1.0 1.7 1.1 77.2 1.4
N 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.0 10.9 0.0 0.5 0.6 9.1 78.7(a) `Take picture’ at house A (b) Another `take picture’ at house A (c) `Brush teeth’ at house B

Fig. 6. Example sensor data sequences obtained in our experimentbecause the orientation of the sensors changes. Fig. 6 (c) shows sensor data
sequences obtained when the participant in house B brushed his teeth. We cansee that the sensor data value of R-I1 suddenly changes several times. Thesechanges were caused by changes of hand posture. We could ﬁnd such sensor datachanges in other ‘brush teeth’ activities in house B. We consider that our methodmodeled the ‘brush teeth’ activity class by using the characteristic changes of
sensor data. As above, even though the hand-worn sensors cannot sense highintensity magnetic ﬁelds emitted by electrical devices, our method may be able
to recognize activities that involve characteristic changes of hand posture, e.g.,
walking and running in addition to tooth brushing.
Recognition with small numbers of sensors. In the above evaluation, we
conﬁrmed that our approach could achieve very high accuracy activity recognition with 20 hand-worn magnetic sensors. However, it is impractical to wear a

290

T. Maekawa et al.

large number of sensors on the hands continuously in our usual daily environment. To reduce the number of required sensors, we investigated which magnetic
sensors on which parts of the hand were eﬀective for recognizing the use of electrical devices by employing our obtained data sets. That is, we ﬁnd sensors
that help us to recognize the use of electrical devices. More speciﬁcally, we ﬁnd
extracted features that help us to discriminate the feature vectors of activity
classes. Once we obtain discriminative features, we then also obtain eﬀective
sensors from the features. For example, when we ﬁnd that the energy feature of
the L-M2 sensor data assists us to recognize many activities, we can say that
the L-M2 sensor is important.
The problem of ﬁnding a discriminative feature is called the feature selection problem. Several studies have found discriminative features by employing
the concept of information gain. The information gain of a feature increases
the better the feature classiﬁes the instances. For more detail, see [22]. In ourcase, an instance corresponds to a feature vector at a time slice. By using theinformation gain, we obtained the ranking of 20 magnetic sensors on hands by
the contribution to the classiﬁcation of instances. We employ the simple ranking
method described below. (1) We compute each feature’s information gain when
distinguishing feature vectors of an activity class from those of other activity
classes by using its feature values in the training data. This permits us to obtainthe measure of the contribution, i.e., the information gain, of each feature to
distinguish the activity class from other classes. We describe the information
gain of the ith feature to distinguish the jth class as gain(fi , Cj ). By applying this procedure to all activity classes, we can obtain the information gainof a feature for each activity class. (2) We compute the sum of the information gain of all activity classes for each feature. We regard the sum as being
an overall measure of the contribution of the feature. We describe the overallmeasure of ith feature as scoref (i) =j gain(fi , Cj ). (3) To obtain a measure of the contribution of the kth sensor, we again compute the summation ofscoref (i) related to the sensor. We can describe the measure of the kth sensor asscores (k) = fi ∈F (k) scoref (i), where F (k) denotes a set of features computedfrom the kth sensor data. (4) We rank the 20 magnetic sensors by scores (k).
The table in Fig. 7 (a) shows the rankings of the top 8 sensors in each housecomputed by employing the above procedure. As described above, in the rankings, the magnetic sensors are listed in descending order of contribution to thefeature vector classiﬁcation. The ‘all’ columns in the table show the rankingsof the top 8 sensors among all 20 sensors. In the rankings, we can see that therank orders of sensors on the right are high in all three houses. This is because
the participants are right-handed. They used electrical devices with their righthands in many activities, and thus the importance of the sensors on the righthand became high. Of the sensors on the right hand, the sensors on the middle
ﬁnger were particularly important. This may be because the middle ﬁnger ispositioned in the center of the hand. We consider that sensors in the centerof the hand tend to be located nearer electrical devices held in the hand thanoutlying sensors such as those on the little ﬁnger. On the other hand, the ‘ring’Recognizing Use of Portable Electrical Devices

291

90

House A
all
ring

House C
all
ring

80

R-M1 R-M2 R-T1 R-T1 R-I1 R-I2
R-M2 R-R2 R-M1 R-M2 R-M1 R-M2
R-R1 R-I2 R-M2 R-I2 R-I2 R-T1
R-R2 R-T1 R-I2 L-R2 R-M2 R-L2
R-I2 L-I2 R-P1 L-L2 R-R1 L-T1
R-T1 L-M2 L-P1 L-I2 R-P1 L-I2
L-I1 L-T1 R-I1 L-M2 R-L1 L-R2
L-M1 R-L2 L-R2 R-R2 R-T1 L-L2

F-measure

1st
2nd
3rd
4th
5th
6th
7th
8th

House B
all
ring

70

60

50
F-measure (all)F-measure (ring)401(a) Ranks of sensors

2

3

4
5
#sensors

6

7

8

7

8

90

80

80

70

70

F-measureF-measure(b) House A
90

60

50

60

50
F-measure (all)F-measure (ring)F-measure (all)F-measure (ring)

40

40
1

2

3

4
5
#sensors(c) House B

6

7

8

1

2

3

4
5
#sensors

6(d) House C

Fig. 7. (a): Rankings of magnetic sensors in each house. The table shows the rankingsof the top-8 sensors of all 20 sensors and the rankings of just 10 sensors on hand
positions where common forms of finger rings are attached. (b), (c), (d): Transitions of
the overall F-measure when we increase # sensors in descending order of the rankings.)in each house. (F -measure = 2·precision·recall
precision+recall

columns in the table in Fig. 7 (a) show the top 8 sensors in the ranking of just
10 sensors at hand positions where common forms of ﬁnger rings are worn, i.e.,(L|R)-T1, (L|R)-I2, (L|R)-M2, (L|R)-R2, and (L|R)-L2. That is, we assume thatmagnetic sensors are embedded in ﬁnger ring-form devices. This is a practicalcondition. In these rankings, the sensors on the right hand were also important.
Fig. 7 (b), (c), and (d) show the transitions of the overall F-measure when we
increased the number of sensors in descending order of the rankings in Fig. 7 (a).
For example, when the number of sensors is 2 in the line chart of house A, the y-axis
value of the ‘F-measure (all)’ line indicates the recognition accuracy computed byusing sensor data obtained from only the R-M1 and R-M2 sensors, which are the
top 2 sensors in house A. We use these charts to investigate the required numberof sensors. From the charts, we ﬁnd that we can achieve high accuracies equaling
those obtained with 20 sensors with only about seven sensors. The accuracies werestill high (over 75%) with only three sensors in houses A and C. The right part ofTable 3 shows the detailed recognition accuracies when we used the sensor data ofthe top 4 sensors. Even though we used only four sensors, the average and overall

292

T. Maekawa et al.

accuracies decreased by only about 5 to 10% compared with those obtained whenusing all 20 sensors. As shown in the charts in Fig. 7 (b), (c), and (d), when welimit the sensor positions to common ﬁnger ring positions, the accuracies decreased
somewhat. However, with only four or ﬁve sensors, we could achieve accuracies ofover 75% in houses A and C. Here, the accuracies in house B are not stable because
the amount of training data may be insuﬃcient.
The above results show that we can achieve the highly accurate recognitionof the use of portable electrical devices with small numbers of magnetic sensors.
Strictly speaking, the importance of the sensor positions was diﬀerent in eachhouse as shown in the table in Fig. 7 (a). However, in any environment, by simply
attaching several sensors to a dominant hand, we consider that we can achieve
fairly good accuracy. (In fact, by simply using the R-M1, R-M2, R-I1, and R-I2
sensors, we could achieve 76.8, 67.6, and 82.7% overall F-measures in houses A,
B, and C, respectively.) In particular, attaching sensors to the middle ﬁngers
may signiﬁcantly improve the accuracy.

7

Conclusion

In this paper, we proposed a new method that recognizes the use of portableelectrical devices with hand-worn magnetic sensors. In modern societies, welive with large numbers of electrical devices. Many studies have attempted todetect/recognize the use of electrical devices because this would allow us to recognize various high-level activities. The method proposed in this paper can recognize the use of portable electrical devices that are not connected to the homeinfrastructure without using any sensors attached to the devices in both indoor
and outdoor environments. In this paper, we evaluated our recognition method
in real environments and achieved very high accuracies. We also conﬁrmed experimentally that we could achieve highly accurate recognition with small numbersof hand-worn sensors. As part of our future work, we will attempt to recognize
simple low-level activities that do not involve the use of electrical devices, suchas walking and running, with hand-worn magnetic sensors because we found in
our experiment that our method may be able to recognize activities involving
characteristic hand movements. This is because magnetic sensors are aﬀected by
the earth’s magnetism, and so the hand-worn sensors can capture characteristic
hand movements. This permits us to recognize both the use of electrical devices
and low-level activities with only hand-worn magnetic sensors.

References
1. Bao, L., Intille, S.S.: Activity recognition from user-annotated acceleration data.
In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 1–17.
Springer, Heidelberg (2004)
2. Blum, M., Pentland, A.S., Troster, G.: Insense: Interest-based life logging. IEEE
Multimedia 13(4), 40–48 (2006)
3. Cohn, G., Gupta, S., Froehlich, J., Larson, E., Patel, S.N.: GasSense: Appliancelevel, single-point sensing of gas activity in the home. In: Floréen, P., Krüger,
A., Spasojevic, M. (eds.) Pervasive Computing. LNCS, vol. 6030, pp. 265–282.
Springer, Heidelberg (2010)

Recognizing Use of Portable Electrical Devices

293

4. The Swedish Confederation for Professional Employees, http://www.tco.se/
5. Froehlich, J.E., Larson, E., Campbell, T., Haggerty, C., Fogarty, J., Patel, S.N.:
Hydrosense: Infrastructure-mediated single-point sensing of whole-home water activity. In: Ubicomp 2009, pp. 235–244 (2009)
6. ICNIRP Guidelines. Guidelines for limiting exposure to timevarying electric, magnetic, and electromagnetic fields (up to 300 GHz). Health Physics 74(4), 494–522(1998)
7. Huynh, T., Schiele, B.: Towards less supervision in activity recognition from wearable sensors. In: Int’l Symp. on Wearable Computers, pp. 3–10 (2006)
8. Kim, Y., Schmid, T., Charbiwala, Z.M., Srivastava, M.B.: ViridiScope: design andimplementation of a fine grained power monitoring system for homes. In: Ubicomp
2009, pp. 245–254 (2009)
9. Lenz, J.E.: A review of magnetic sensors. Proceedings of the IEEE 78(6), 973–989(1990)
10. Lester, J., Choudhury, T., Borriello, G.: A practical approach to recognizing physical activities. In: Fishkin, K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) PERVASIVE 2006. LNCS, vol. 3968, pp. 1–16. Springer, Heidelberg (2006)
11. Lester, J., Choudhury, T., Kern, N., Borriello, G., Hannaford, B.: A hybrid discriminative/generative approach for modeling human activities. In: IJCAI 2005,
pp. 766–772 (2005)
12. Lukowicz, P., Ward, J., Junker, H., Stäger, M., Tröster, G., Atrash, A., Starner, T.:
Recognizing workshop activity using body worn microphones and accelerometers.
In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 18–32.
Springer, Heidelberg (2004)
13. Maekawa, T., Yanagisawa, Y., Kishino, Y., Ishiguro, K., Kamei, K., Sakurai, Y.,
Okadome, T.: Object-based activity recognition with heterogeneous sensors on
wrist. In: Floréen, P., Krüger, A., Spasojevic, M. (eds.) Pervasive Computing.
LNCS, vol. 6030, pp. 246–264. Springer, Heidelberg (2010)
14. Maekawa, T., Yanagisawa, Y., Kishino, Y., Kamei, K., Sakurai, Y., Okadome, T.:
Object-blog system for environment-generated content. IEEE Pervasive Computing 7(4), 20–27 (2008)
15. Mayol, W.W., Murray, D.W.: Wearable hand activity recognition for event summarization. In: Int’l Symp. on Wearable Computers, pp. 122–129 (2005)
16. Patel, S.N., Robertson, T., Kientz, J.A., Reynolds, M.S., Abowd, G.D.: At the
flick of a switch: Detecting and classifying unique electrical events on the residential power line. In: Krumm, J., Abowd, G.D., Seneviratne, A., Strang, T. (eds.)
UbiComp 2007. LNCS, vol. 4717, pp. 271–288. Springer, Heidelberg (2007)
17. Philipose, M., Fishkin, K.P., Perkowitz, M.: Inferring activities from interactions
with objects. IEEE Pervasive Computing 3(4), 50–57 (2004)
18. Ravi, N., Dandekar, N., Mysore, P., Littman, M.L.: Activity recognition from accelerometer data. In: IAAI 2005, vol. 20, pp. 1541–1546 (2005)
19. Tapia, E.M., Intille, S.S., Larson, K.: Activity recognition in the home using simple
and ubiquitous sensors. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004.
LNCS, vol. 3001, pp. 158–175. Springer, Heidelberg (2004)
20. van Kasteren, T., Noulas, A., Englebienne, G., Krose, B.: Accurate activity recognition in a home setting. In: Ubicomp 2008, pp. 1–9 (2008)
21. Welch, L.R.: Hidden markov models and the baum-welch algorithm. IEEE Information Theory Society Newsletter 53(4) (2003)
22. Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, San Francisco (2004)