IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

Regular Paper

A Ubiquitous Power Management System to Balance
Energy Saving and Response Time based on Devicelevel Usage Prediction
Hua Si,†1 Shunsuke Saruwatari,†1
Masateru Minami†1 and Hiroyuki Morikawa†1Power conservation has become a serious concern during people’s daily life.
Ubiquitous computing technologies clearly provide a potential way to help us
realizing a more environment-friendly lifestyle. In this paper, we propose aubiquitous power management system called Gynapse, which uses multi-modal
sensors to predict the exact usage of each device, and then switches their power
modes based on predicted usage to maximize the total energy saving under theconstraint of user required response time. We build a three-level Hierarchical
Hidden Markov Model (HHMM) to represent and learn the device level usage
patterns from multi-modal sensors. Based on the learned HHMM, we develop
our predictive mechanism in Dynamic Bayesian Network (DBN) scheme to precisely predict the usage of each device, with user required response time underconsideration. Based on the predicted usage, we follow a four-step process tobalance the total energy saving and response time of devices by switching theirpower modes accordingly. We use PlaceLab data set to evaluate Gynapse, andthe preliminary results demonstrate that Gynapse has the capability to reduce
power consumption while keeping the response time not exceed user requirement, which provides a complementary approach to previous power management systems.

1. Introduction
Rising global energy demands, increasing costs and limitations on natural resources have raised the concerns about energy conservation. As reported by
Japanese government1) , the energy consumption in Residential & Commercial
sector has increased 40% since 1990, among which the most consumed energy iselectricity by various home electronic devices2) . Ubiquitous computing technologies provide the potential to reduce power consumption. For instance, the lights,†1 Research Center for Advanced Science and Technology, The University of Tokyo

1234

heating and other devices all sit on a ubiquitous in-house network. Lights canbe automatically controlled by sensors that measure the brightness in the room,and they will switch oﬀ altogether if no one is in the room. The temperatureof the living room air conditioning or ﬂoor heating will change according to thenumber of people in the room. We can also check if the house is carbon neutral
by seeing how much power is generating from solar panels and how much is used.
In this paper, we will discuss the power saving potential of ubiquitous computing
technologies in an indoor environment.
People have tried a lot of methods to reduce power consumption in indoor
environments. The most common way is replacing with energy eﬃcient devices.
However, replacement cannot solve all the problems. For instance, even if theresident replaces a 60W bulb with a 30W energy eﬃcient one, it will still waste a
lot of energy if he leaves the bulb ON for 24h/7d. To eliminate such waste, people
try an alternative way to switch lights oﬀ or turn devices into low power mode
when not in use3) , which we call “power mode switching” indistinctly herein.
This method has drawn special attention from researchers of ubiquitous computing, and shown promising results4)–11) . These systems try to automaticallyswitch power modes of devices with sensors and controllers installed in indoor
environments. We will follow their researches and focus on “ubiquitous powermode switching” systems in this paper.
According to our research, we believe the following requirements are important
for a practically eﬀective power mode switching system:• A system should have the capability to adaptively handle multiple devices
for complicated human behaviors in diﬀerent situations. Because of the complexity of human behavior, a resident usually uses diﬀerent combinations ofdevices in diﬀerent situations. Therefore, it is natural to require a system
to be able to adaptively coordinate the power mode switching of multiple
devices for complicated behaviors in diﬀerent situations.
• The system should have the capability to precisely predict the exact usageof each device. The key of power mode switching is to proactively determine
the future usage of a device, and switch its power mode accordingly. Hence,
precise usage prediction is crucial for such a system. In addition, since we
can only switch power mode of each device, the usage prediction should alsoc 1977 Information Processing Society of Japan
⃝

1235

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction

be at device level. As a result, a system must be able to precisely predict theusage at device level.
• The system should have the capability to balance the response time of devices
and their total energy saving. For instance, although a PC can save energy insleeping mode, the resident may feel frustration with its long wake-up time.
Therefore, a system must be able to save energy while keeping the responsetime short enough.
Previous power mode switching systems4)–11) usually make power mode switching according to pre-deﬁned “rules” or the resident’s locations. Although theysuccessfully meet one or two requirements above, none of them can fulﬁll all threerequirements. In this paper, we propose a ubiquitous power management systemcalled Gynapse, which uses multi-modal sensors to predict the exact usage ofeach device, and then switches their power modes based on predicted usage tomaximize the total energy saving under the constraint of user required response
time. To our best knowledge, it is the ﬁrst system that fulﬁlls all the requirements
above. Gynapse consists of three important components:• A probabilistic model to learn residents’ usage patterns at device level frommulti-modal sensors. We build a three-level Hierarchical Hidden Markov
Model (HHMM)12) to represent multiple residents and their device usage,
and use Forwards-Backwards (FB) and Expectation-Maximization (EM) algorithms13) to learn the parameters. The sensor data, such as RFID readingfrom keyboard and current data from power lines, are arranged as vectors
to train HHMM. This model provides the capability to adaptively handle
multiple devices for complicated behaviors in diﬀerent situations.
• A predictive mechanism to forecast the usage probability of multiple devices
in the future. Based on the learned three-level HHMM, we introduce two variable to represent device’s wake-up time and user’s required response time, anddevelop our predictive mechanism in Dynamic Bayesian Network (DBN)14)
scheme. This mechanism provides the capability to precisely predict theusage of each device, and also takes the response time into consideration.
• A control framework to maximize the energy saving under the constraint ofuser required response time. With the predicted usage probability of eachdevice, we follow a four-step process to calculate the total probability and

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

energy saving of devices, and switch their power modes according to thescenario with the highest energy saving and probability. This frameworkputs everything together and balances the response time of devices and their
total energy saving.
To conduct our analysis, we use sensor data from MIT PlaceLab15) to implement and evaluate Gynapse. We totally obtain 24 days of data, within which weuse 9 days to train the three-level HHMM, and 14 days to verify the system. After learning the parameters of our probabilistic model, the predictive mechanism
correctly predicts the device usage for about 90%. Based on the usage prediction,
the control framework successfully balances the response time and energy saving,
and achieves an average 11% power saving of 14 days.
The succeeding sections of this paper are organized as follows: Section 2 reviews
the previous researches on power mode switching systems. Section 3 analyzes
the technical problems we are going to solve. Section 4 describes the design ofGynapse. In section 5, we evaluate our system with PlaceLab data and discuss
the preliminary results. Section 6 concludes the paper ﬁnally.
2. Related Works
In this section, we will review the previous power mode switching systems
according to the three requirements mentioned in section 1.
The most straightforward way of power mode switching is automatic poweroﬀ based on pre-deﬁned “rules”. Such systems use infrared or motion sensors
to detect the existence/absence of residents, and then turn on/oﬀ the lights or
air-conditioners through Home Energy Management System (HEMS)4),5) . Their
motivation is the simplicity to build a rule, such as “If no user in the room fora time-out period, then turn oﬀ the lights.” However, the human behaviors are
so complicated that cannot be completely described with such simple rules, e.g.,
if one sits still in a chair when reading, the lights may be incorrectly turned oﬀ
after a motion time-out period. Additionally, as the number of rules increases,
it may become diﬃcult to coordinate rules in diﬀerent situations. For instance,
one rule may say “If no user sits in front of TV for a time-out period, then turnoﬀ TV”; whereas, another rule may say “If the user is cooking in kitchen, thenturn on TV”, because he usually watches it when cooking. To eliminate suchc 1977 Information Processing Society of Japan
⃝

1236

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction

conﬂicts, people have to deﬁne complex rules to describe the situations, which
digress from the original standpoint of simplicity. As a result, such “rule-based
switching” systems do not fulﬁll the ﬁrst requirement, since they cannot properly
handle multiple devices for complicated human behaviors in diﬀerent situations.
To solve the problems of “rule-based switching” systems, researchers have tried
probabilistic models for more eﬃcient power management. The ﬁrst work is
Neural Network House6),7) , which builds a neural network to predict the mobilityof residents between diﬀerent “lighting zones”, and switch the power setting oflights accordingly. A similar work is MavHome project8) , which predicts theresident’s location together with his most likely path based on an informationtheoretical framework, and pro-actively switches the power mode of devices along
this future route. Harris and Cahill9),10) introduce the concept of context-aware
power management, and concentrate on switching power modes of a desktop PC
according to predicted usage. They build a Dynamic Bayesian Network (DBN)
for usage prediction based on user’s proximity to PC. Although people argue
that probabilistic prediction rarely achieve 100% accuracy, it can be improvedby integrating high-level information, such as user feedback16)–18) .
A more serious drawback of previous probabilistic power mode switching systems is they are all based on the prediction of resident’s coarse-grained location
instead of each device’s exact usage. Therefore, they leave two problems unsolved:1) they imply the device location is ﬁxed, so they can predict usage based on resident’s proximity (location) to device. However, for devices such as TV, which
are controlled via remote, proximity does not work. Furthermore, they cannot
predict proximity to remote, because it is movable. 2) They only discuss thesituation that one (kind of) device at one coarse-grained location, such as thelight in a room. However, it is not unusual that a TV, a DVD player and a lightare in the same room. In this case, previous works cannot decide which device
to use, even if they can predict the resident will enter this room. Therefore,such “coarse-grained location-based switching” systems do not fulﬁll the second
requirement, because they cannot predict the exact usage of each device.
People may argue that we can solve the preceding problems, if we have a highly
ﬁne-grained location system that pinpoints every device no matter movable ornot. Unfortunately, no such systems exist at current stage. However, researchers

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)of activity recognition provide an alternative way to model the device level usage
by using multi-modal sensors. In 19), 20), they attach RFID tags to devices
such as TV remote and washer, and build probabilistic models such as Hidden
Markov Model (HMM) or Dynamic Bayesian Network (DBN) to infer the activities in which the devices are used. 15), 21) use more multi-modal sensors torecognize activities in addition to RFID tags. However, since the objective of
these researches is to infer what activity it is given the observed sensor data, notpredict the activity and device usage in the future, they do not fulﬁll the second
requirement too.
As we are aware, no existing power mode switching system is based on the usage
prediction at device level. In this paper, we try to build such a system by using
multi-modal sensors, since it is more realistic than a ﬁne-grained location-based
system. We must make it clear that our system is complementary rather thancompetitive to previous coarse-grained location-based systems. On one hand, we
can use them jointly to increase accuracy. For example, we can predict the roomto enter from location system, and predict the device to use from multi-modal
sensors. On the other hand, we can use device level prediction independently.
For instance, no matter a resident’s location is in living room or kitchen, we canalways capture his usage of TV from RFID tag attached to the remote. In thispaper, we will learn and forecast the exact usage of each device from multi-modal
sensors, and switch their power modes based on the usage prediction at device
level.
The third requirement comes from a major frustration of previous power modeswitching systems, which is the users have to wait for a long wake-up time before
using a device11),22) . To solve this problem, Harle and Hopper11) classify theelectronic devices into three “wake-up time” categories, and optimize their powermode switching based on a location-aware system in an oﬃce building. However,they have the same problem as other location-based power management systems,
and cannot address the problem such as remote-controlled devices or multiple
devices at the same location.
From the discussion above, we can ﬁnd that no previous systems fulﬁll all
three requirements. To complement the previous researches, we propose Gynapse,
which uses multi-modal sensors to predict the exact usage of each device, and thenc 1977 Information Processing Society of Japan
⃝

1237

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction

switches their power modes based on predicted usage to maximize the total energy
saving under the constraint of user required response time. In the following
section, we will analyze the technical problems need to be solved by Gynapse.
3. Problem Statements⋆1 For clarity, we deﬁne resident’s direct interaction with a device as a low level “action”, anddeﬁne his goal of a sequence of actions as a high level “activity”.
⋆2 As projected by the National Institute of Population and Social Security Research, becauseof the falling birthrate and the aging population, the family structure in Japan will haveinﬂuential change by 2010: 31.2% will be single live, 20.1% will be only husband and wife,
27.9% for parents and kids, 9% for single-parent and kids, and 11.8% for others25) .

IPSJ Journal

Vol. 18

No. 4

Device 1

1234–1249 (Apr. 1977)

Without
Restriction

Working

Low Power

Twu
TRS
With
Restriction

Working

Low Powertt1

Working

Low Power

Twu,2Device 2

As we explained in section 1, three components are important for Gynapse:
1) a probabilistic model to learn residents’ usage patterns at device level, 2) a
predictive mechanism to forecast the usage probability of multiple devices in the
future, 3) a control framework to maximize energy saving under the constraintof user required response time. We will analyze the technical challenges of eachcomponent, and formulate the problems we are going to solve.
3.1 Challenges for Probabilistic Model
The main challenge for probabilistic model derives from the fact that hierarchical structure exists in human behaviors23),24) . For instance, during a high level
activity such as “make breakfast”, the resident may repeat low level actions⋆1 ,such as “use coﬀee maker” and “use toaster”; or during the activity of “work inoﬃce”, he may repeat actions of “use PC” and “use lamp”. Hence, if the resident
is “using coﬀee maker”, it is more possible for him to “use toaster” rather than“use PC” for the next step. This kind of sequences (or transitions) of actions
and activities are actually the “patterns” of resident’s device usage. To learn
them, the probabilistic model must properly represent the hierarchical structure
in human behaviors and the transitions between actions and activities.
Another challenge is multiple residents in one family. Although one-personfamily will become the largest single category in Japan by 2010 with a percentage of 31.2%, multi-person family will still be the mainstream⋆2 . Therefore,
probabilistic model need another hierarchy to represent the status of residents.
3.2 Challenges for Predictive Mechanism
After learning the usage patterns, we can predict the usage probability of multi-

Twu

Twu,1

t2

Time

T'1

Fig. 1 Diﬀerent wake-up times of devices

Working

Low Power
T

T'2

T2

Time

Fig. 2 Required response time

ple devices in the future. There are three challenges for the predictive mechanism.
The ﬁrst one is it must be able to handle diﬀerent wake-up times. Figure 1 depicts an example. Two devices have diﬀerent wake-up times, TW U,1 and TW U,2 ,respectively. If both of them are awoken at time t, they can only be used after
t1 =t + TW U,1 and t2 =t + TW U,2 , respectively. Therefore, if a system wants tomake wake-up decisions at time t, it has to predict the usage probability of device 1 at time t1 , and that of device 2 at time t2 , which is deﬁned as wake-up
probability of device i at time t, pt,i .
The second challenge for predictive mechanism is it must consider an acceptable
response time ⋆3 for the resident. An example is illustrated in Fig. 2. A service
request comes at time T . If the device is awoken at that time, the resident cannot
use it until T2 =T + TW U . However, if the resident restricts the response time no′′
more than TRS =T2 − T , then the device must be awoken at least T1 , which is(TW U − TRS ) before the service request at T ⋆4 . As a result, if we want to make′a wake-up decision at T1 like t in Fig. 1, the wake-up probability pT ′ is actually′′1

the device usage probability at T1 + (TW U − TRS ) instead of T1 + TW U .
The third challenge is improving the accuracy of prediction. Because the predicting algorithm itself rarely achieve 100% accuracy, some high-level information
should be integrated to improve it.
3.3 Challenges for Control Framework
After predicting the wake-up probability of devices, we can control power mode
⋆3 Response time is deﬁned as the duration between a service is requested and a device is′ready to provide it, such as T2 − T or T2 − T in Fig. 2⋆4 The practical upper limit of TRS is TW U , since the user will not accept a device to response
longer than its wake-up time.

c 1977 Information Processing Society of Japan
⃝

1238

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction

switching to maximize the total energy saving under the constraint of user required response time. As explained in section 3.2, the user required response
time, TRS , has been considered when predicting wake-up probability pT ′ . If this1′

probability is less than 0.5, which means the device should not be awoken at T1 ,then the device has to postpone its wake-up, and the response time will exceeduser’s requirement TRS . Therefore, the restriction that the response time should
not exceed TRS actually becomes that the wake-up probability pT ′ should not
1
fall below 0.5. As a result, the challenge for control framework becomes choosing a combination of devices that will save maximum energy after power modeswitching, under the constraint that the wake-up probability of this combination
is higher than a threshold.
4. Design of Gynapse
To solve the challenges in section 3, we design a system called Gynapse, whichlearns and predicts the exact usage of each device, and then switches their power
modes accordingly to maximize the total energy saving under the constraintof user required response time. A simpliﬁed example of Gynapse is shown in
Fig. 3⋆1 .
We obtain power status of PC, light, tea-maker, and microwave from current
sensors, which are denoted as WK/LP for working/low power modes. Obviously,
power consumption depends on device usage, which is closely correlated withsensors, such as RFID tag on microwave or Object Movement (OM) sensor on
tea-maker (circles in Fig. 3). Therefore, we can infer the resident’s activities anddevice usages from multi-modal sensors. As shown in Fig. 3, after using PC inoﬃce, the resident enters kitchen and turns on light at 20; then he heats waterwith tea-maker and warms food with microwave at 25 and 35 respectively; at 80,
he leaves kitchen and works on PC again. With this analysis, we know the PC is
not used from 20 to 80, so it could be switched into low power mode for savingenergy (shaded bar in Fig. 3).
To achieve this goal, we must proactively switch power mode of PC according
to its predicted usage: if we predict the user will head for kitchen to make a⋆1 We rescale time line and ignore details of sensor data that will not detract our discussion.

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

PC in
officeRFID onkeyboard
Motionin officeLight inkitchen
Motion in
kitchenTeamakerOM onteamaker
Microwave
RFID onmicrowave

ＷＫ
ＬＰ0

10

20

30

40

50

60

70

80

90

100

0
0
1

10

20

30

40

50

60

70

80

90

100

0
0

10

20

30

40

50

60

70

80

90

100

10

20

30

40

50

60

70

80

90

100

10

20

30

40

50

60

70

80

90

100

10

20

30

40

50

60

70

80

90

100

10

20

30

40

50

60

70

80

90

100

10

20

30

40

50

60

70

80

90

100

10

20

30

40

50

60

70

80

90

100

1

ＷＫ
ＬＰ0
1
0

ＷＫ0
ＬＰ0
1
0

ＷＫ0
ＬＰ0
1

0
0

Gynapse
Forecaster3

Controller

2

Data
Aggregator

Device( S, Twu, E )1Fig. 3 An example of Gynapse

4

Ubiquitous Computing Environment
Sensors

UI

DevicesFig. 4 System architecturesnack, we shall turn PC into low power mode; if we predict he will return towork, we shall turn PC into working mode. Therefore, we need a functional partin Gynapse to predict device usage. Another consideration is, the user may feel
frustration because the long wake-up time of PC. Therefore, we need a functionalpart in Gynapse to take response time into consideration.
With these considerations, we design Gynapse’s architecture as in Fig. 4. We
assume a user interface (UI), sensor and control infrastructure exists in an indoor ubiquitous computing environment, which could be used to collect user
feedback, detect human activities and switch power mode of devices. Gynapse
consists of one adjunct database of device information, and three functional parts:
Data Aggregator, Forecaster, and Controller. Solid arrows are main data ﬂows.
Data Aggregator receives information from diﬀerent types of sensors/devices (arrow 1), normalizes their values, and builds them into a time series of vectors()⃗ t = s1t , s2t , . . . , sMOt , where M is the number of sensors and t denotes time.
After receiving these vectors (arrow 2), Forecaster builds a three-level Hierarchical Hidden Markov Model (HHMM) to learn the usage pattern of devices,and forecast their usage probabilities through the predictive mechanism. With
the predicted probabilities (arrow 3), Controller makes decisions of power modeswitching to maximize the total power saving under response time constraint.
These decisions (arrow 4) are then sent to the ubiquitous computing infrastructure to switch devices. If Gynapse makes any wrong decisions, user feedback can
be obtained through user interface. The adjunct database provides the necessary
information of current status (S), wake-up time (TW U ), and power saving (E) of
each device.

c 1977 Information Processing Society of Japan
⃝

1239

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction
R

First Level

In the following sections, we will discuss the details of probabilistic model andpredictive mechanism in Forecaster, and control framework in Controller.
4.1 The Probabilistic Model
Before predicting the usage probability of devices, Forecaster must build aprobabilistic model to learn the usage patterns of them. We use a three-level
Hierarchical Hidden Markov Model (HHMM) to represent and learn residents
status and device usages, because its hierarchical nature easily characterizes natural hierarchies in human activity and allows for model reusability and more
eﬃcient learning12),26) . Our contribution here is applying HHMM to model thedevice usage in a power management system, instead of developing a new extension of HHMM. Therefore, we focus on explaining the model of single resident
and its generalization to multiple residents, rather than discussing the details oflearning algorithms.
Figure 5 illustrates a model, whose left part with numbered arrows corresponds to our example in Fig. 3. Five diﬀerent types of nodes are in this model:
R, I, P , E, and O. The R node at the ﬁrst level is Root state representing
the status of residents at home. For single resident, it is state (1) if he is athome, otherwise it is state (0). The I nodes at the second level are Internal
states, which represent activities of the resident, such as “make snack”, or “workin oﬃce”. They may have an arbitrary number of P nodes. The P nodes at
the third level are Production states, which represent the device status, such as“Tea-maker (TM) is in use”, or “microwave (MW) is not in use”. Production
state is the only one within HHMM that can emit observations, O nodes, which
are the vectors sent by Data Aggregator. The E nodes are End states, which
exist only to signal the horizontal transition is ended, and a vertical transition
to upper level is needed.
Arrows between nodes represent probabilistic dependencies: solid arrows aretransitions between hidden states of HHMM, and dotted line arrows are dependencies between Production states and observations. There are two kinds of statetransitions: 1) horizontal transition at the same level, which means the resident
is going to use another device or to another activity; 2) vertical transition between diﬀerent levels, which means the resident start or ﬁnish using devices inan activity. For instance, the resident “turns oﬀ light” after “using microwave”

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

3

Second Level
Make snack

Work in
officeI

Activity

I

Activity

I

I

Activity

I

E

P

Activity

E

I

10
11

4
9

5

P

6

P

TM

Observations

2
1

8

Third Level

O

7
MW

O

P

E
Light

O

P

E

P

E

P

P

O

O

E

P

P

P

O

O

O

E

PC

O

O

O

Fig. 5 State transition diagram of three-level HHMM. Shaded nodes are observed; the remaining nodes are hidden. The numbered arrows show the sequence of state transitions.

(7), a transition to End state (8) means he has ﬁnished “making snack” anda transition back to the second level (9) is needed. Then he goes to “work inoﬃce” and “use PC”, which are represented by transition (10) and (11). Thismodel clearly represents the hierarchical structure in human behaviors and thetransitions between device usages and activities.
When multiple residents are at home, it becomes a little more complex. Ifthey always do the same activities, such as “watch TV” together, it will not be
a problem. If they do diﬀerent activities at the same time, we have to represent
them separately in the model. As explained before, R node represents the statusof residents at home. If there are two people, we add one dimension to R node,so the state (1,1) represent both of them are at home. It is the same for I and P
nodes. For instance, one state of I node will become (make snack, watch TV) torepresent their activities respectively. Of course, as the dimension increases, thenumber of state will also increase. Fortunately, it will not increase exponentially,
because the residents in a family tend to do the same activity for most of thetime, such as “have dinner” or “watch TV” together.
After solving the representation challenges in section 3.1, we now focus onlearning usage patterns, which is actually learning the probabilistic dependencies: horizontal transition probabilities, vertical transition probabilities, and observation probabilities. We follow the notation in 12) to deﬁne them. The Root,Internal, Production and End states in Fig. 5 are uniformly represented by qid ,where d ∈ [1, 2, 3] denotes the hierarchy level and i is the state index relative toits parent. E.g., qi2 means an Internal state at the second level. The probabilityd−1of state q d−1 vertically transitioning to its children qid is speciﬁed as π q (qid ).
c 1977 Information Processing Society of Japan
⃝

1240

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction

The probability of state qid making a horizontal transition to state qjd is writtend
as aqij .

Hence, for Root and each Internal state, vertical and horizontal transiddtion probabilities are deﬁned as the vector Πq and the matrix Aq respectively.
Production states at the third level q 3 are the only type can emit observations.
Therefore, the observation probabilities of Production nodes is represented as the
3vector B q , which deﬁnes the probability of state q 3 producing observation O = k()
3
as bq (k) = P O = k|q 3 ⋆1 . The model parameters are denoted in a compact
form as λ = (A, B, Π).
We use the forwards-backwards and expectation-maximization algorithms in13) to learn the parameters from historical data, which reduces the time complexity to O(T ), compared with O(T 3 ) in the original paper of 12). The learned
parameters λ = (A, B, Π) are used to predict the probability of device usage via
the predictive mechanism.
4.2 The Predictive Mechanism
After learning the usage patterns, Forecaster predicts the usage of multiple
devices in future through the predictive mechanism. In order to show the temporal relationship more clearly, we redraw the three-level HHMM as a Dynamic
Bayesian Network (DBN)13),14) in Fig. 6, which is identical to the state transition
diagram in Fig. 5. Although Fig. 5 emphases the parent-children relationshipbetween states, Fig. 6 highlights the temporal sequence of them.
Figure 6 shows the three-level HHMM as time slices. At time slice t, the()
d⃗ t = s1t , s2t , . . . , sM
observation is Ot ; the state at level d is denoted as Qt , so theRoot, Internal, and Production states are Q1t , Q2t , and Q3t respectively. Ftd is abinary indicator that is 1 if it has entered an End state; otherwise it is 0. The
downward arrows between the Q variables represent a state “calls” its children.
The upward arrows between the F variables enforce the fact that a higher-levelstate can only change when lower-level one is ﬁnished.
The example in Fig. 5 is also shown in Fig. 6. Suppose at t = 1, Q11 represents
the resident is at home, Q21 represents he is “making snack”, and Q31 represents
he is “using microwave”. Then at t = 2, Q12 and Q22 keep unchanged, while Q32

F11

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

F31

at homeat home

Q11

Q21

Q31

F12

make
snack

Q12

make
snack

F22

Q22

use micro
-wave

F13

Q13

not use
light

O1

use
PC

F32

QT+1

QT2

QT+2

QT+12

QT+22
FT+23
QT+23

OT+1
t=T

FT+22

FT+13
QT+13

OT
t=3

FT+21
1

FT+12

FT3
QT3

O3

FT+11
1

FT2

F33

Q33

t=2

FT1
QT1

Q32

O2
t=1

work in
office

F23

Q23

OT+2
t=T+1

t=T+2

Fig. 6 Dynamic Bayesian Network of three-level HHMM. Shaded nodes are observed; theremaining nodes are hidden. Qdt is the state at time t, level d; Ftd = 1 if the HHMM
at level d has ﬁnished (entered its end state), otherwise Ftd = 0.

changes its state to “turn oﬀ light”. Since it is the end of “make snack”, F23
becomes 1, which means it has ﬁnished and Q22 can change state for the nexttime slice. At t = 3, Q13 is still unchanged, Q23 changes to “work in oﬃce”, andQ33 changes to “use PC”.
Given the learned parameters λ = (A, B, Π) in section 4.1, we ﬁrst explain how
to predict the states at the next one time slice, and then generalize to next N
time slices.
d
Suppose we are at t − 1, and know the states Qdt−1 and Ft−1
, where d ∈ [1, 2, 3].
dd
We try to predict the states at t, Qt and Ft . We explain the Production, Internaland Root states separately.
Production state (the third level): Production state Q3t follows a Markov
chain with parameters determined by higher-level states Q1:2
t , which we denoteas k for brevity.
If Q3t−1 does not enter an End state, the value of Q3t should be drawn from3
to be 1 to signal
horizontal transition probability A. Otherwise, it turns Ft−1
3
it is ﬁnished, and draws the value of Qt from vertical transition probability Π.
Formally, we can write the probability that Q3t will be in state j as follows:
{
3
( 3
)
aqij
if f = 0
3
3
1:2
(1)
P Qt = j|Qt−1 = i, Ft−1 = f, Qt = k =2
π q (qj3 ) if f = 1
3⋆1 This can be considered as if q 3 is in the state “using PC”, then we can observe readingfrom RFID tag on keyboard.

F21

at home2where aqij and π q (qj3 ) are the horizontal and vertical transition probabilities
explained in section 4.1. For clarity, i, j ̸= End state here, because the End statec 1977 Information Processing Society of Japan
⃝

1241

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction

is indicated with Ft3 as:()
q3
3
P Ft3 = 1|Q1:2
t = k, Qt = i = ai,end(2)Internal state (the second level): Similar to Production state, Internal
2state Q2t follows a Markov chain with parameters determined by Q1t , and Ft−1
speciﬁes whether we should use the horizontal or vertical transition probability.
3
The diﬀerence is that we now also get a signal from lower-level Ft−1
: if it has
ﬁnished, we are free to change Internal state, otherwise we must remain in thesame state. Formally, we can write the probability that Q2t is in state j as: δ(i, j)
b=0
(
) 
2
3
2
P Q2t = j|Q2t−1 = i, Ft−1
= b, Ft−1= f, Q1t = k =aqij
b = 1 and f = 0
 q1 2π (qj ) b = 1 and f = 1(3)
where δ(i, j) = 1 if i = j, otherwise δ(i, j) = 0. Ft2 should turn to 1 only if Q2t is“allowed” to enter an End state. Formally, we can write this as follows:
{
( 2
)
0
if b = 01
2
3
P Ft = 1|Qt = k, Qt = i, Ft = b =(4)
q2
ai,end if b = 1
Root state (the ﬁrst level): The Root state Q1t diﬀers from Internal statesin that it has not parent to specify which distribution to use. The equations are
the same as above, except we eliminate the conditioning on Q1t = k.
Following Eq. (1) – (4), we can obtain the probabilities of each state at t,
P (Qdt ) and P (Ftd ), given the parameters λ = (A, B, Π) and the states Qdt−1 andd. Now we generalize it to next N time slices.
Ft−1
As explained in section 3.2, we must consider wake-up time TW U and user
required response time TRS for predicting. Suppose we are at time T , and needpredict the probability of device usage at T +(TW U −TRS ). Suppose (TW U −TRS )is equal to the length of 2 time slices⋆1 , so our problem becomes predicting theprobability of QdT +2 , given the current states QdT . We follow Eq. (1) – (4) toobtain the probability of QdT +1 at ﬁrst, and then re-use Eq. (1) – (4) to calculate
the probability of QdT +2 based on QdT +1 . For devices with diﬀerent (TW U − TRS ),we need repeat this process for diﬀerent times.
⋆1 For simplicity, we use discrete time Markov chain here, so (TW U − TRS ) is equal to an
integer number of time slices.

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

The third challenge discussed in section 3.2 is the accuracy of prediction. For
instance, Forecaster predicts the resident will “work in oﬃce” after “making
snack”, whereas he may actually go to “watch TV”. Essentially, it is impossible tocompletely eliminate the incorrect prediction. However, we believe the possibility
still exists to improve Gynapse’s accuracy. A potent candidate is integratinguser feedback with predictive mechanism. As shown in 16), 17), the accuracy
can be improved by introducing high-level information, such as user feedback.
It actually involves two separate problems: the ﬁrst is soliciting feedback fromuser. This one has been intensively studied by researchers of computer-humaninterface, such as explicit feedback from mobile touch screens27),28) or implicit
feedback from sensor networks29),30) . With this kind of technologies, we can easily
obtain feedback from users. The second and key problem is integrating feedback
with learning and predicting mechanism in Forecaster. The general approaches
are: 1) treating user feedback as hard constraints to Bayesian Network’s learningalgorithms18),31),32) , or 2) incorporating them in the prior knowledge33) . Since
the correctness of the learned model obviously depends on the training samples,
we use a training sample selection method similar to 18). Suppose we obtain one
feedback to indicate the “make snack→watch TV” transition is correct, while“make snack→work in oﬃce” is incorrect. We replace one set of observation
vectors that represent “make snack→work in oﬃce”, with one set representing“make snack→watch TV” in training samples; we will replace two sets if we get
two feedbacks. As a result, we punish the wrong prediction and reward the user’s
correction, so we can gradually improve prediction accuracy when we re-learn themodel parameters from new training samples.
In this way, we have solved all the challenges in section 3.2. At last, we send
the predicted probabilities of Production states to Controller, since they are thewake-up probabilities of devices.
4.3 The Control Framework
After receiving the predicted wake-up probabilities, Controller switching powermode of multiple devices to maximize the total energy saving under the constraintof user required response time. As explained in section 3.3, since the user required
response time TRS has been considered in predicted wake-up probabilities, theproblem actually becomes choosing a scenario that will save the maximum energyc 1977 Information Processing Society of Japan
⃝

1242

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction

after power mode switching, under the constraint that the wake-up probabilityof that scenario is higher than a threshold. Four steps are necessary.
The ﬁrst step is to calculate the predicted probability of all scenarios. We
use Ii to indicates the usage of device i, where Ii = 1 if it is awoken to working
mode, and 0 if it is keeping in low power mode. Hence, the predicted probabilityof device i can be written as:{pi
Ii = 1Pi (Ii ) =1 − pi Ii = 0
where pi is the wake-up probability received from Forecaster.
If we totally have N devices, the predicted scenario can be represented as(I1 , I2 , . . . , IN ), and its probability is:∏
P (I1 , I2 , . . . , IN ) =Pi (Ii )(5)1≤i≤NAs explained in section 3.3, we choose scenarios whose probability is higher
than 0.5N as the “possible” pool. We use a pool of scenarios instead of the top
one is because the actual scenario more likely falls into a high probability pool,rather than exactly matches the top 1 predicted scenario⋆1 . At the same time, we
also narrow down the number of scenarios that need to be processed at following
steps.
The second step is to calculate the energy saving of selected scenarios. We
use ei to represent the energy saving of device i, which is the diﬀerence of energy
consumption between working and low power modes. Then the energy saving of
scenario (I1 , I2 , . . . , IN ) is:∑ei ∗ (1 − Ii )ES(I1 , I2 , . . . , IN ) =1≤i≤Nwhich means device i can save energy when it is in low power mode (Ii = 0). To⋆1 We can consider a following example: Suppose we predict the top 1 scenario is S1 . If
our prediction is perfect, the actual scenario must match S1 . However, since no existingalgorithms can achieve 100% accuracy, there must be some actual scenarios mismatch the
top 1 prediction. Now we consider an “extreme” pool, which includes ALL the possible
scenarios. In this case, the actual scenario must fall in this pool with 100% accuracy.
This phenomenon has been broadly observed across predicting algorithms. Some numericalresults can be found in 34), 35), where the top 4 or 5 prediction always achieve higher
accuracy than the top 1 prediction.

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

make it comparable, we normalize the energy saving as:∑e i ∗ Ii
ES(I1 , I2 , . . . , IN )
1≤i≤N
g
∑
ES(I1 , I2 , . . . , IN ) ==1− ∑ei ∗ 1ei ∗ 1
1≤i≤N

(6)

1≤i≤N

The third step is to calculate the product of energy saving and predictedprobability for the selected pool. The product of scenario (I1 , I2 , . . . , IN ) can becalculated by multiplying Eq. (5) and Eq. (6) as:
g 1 , I2 , . . . , IN ) ∗ (1 + P (I1 , I2 , . . . , IN ))U (I1 , I2 , . . . , IN ) = ES(I(7)
We choose the scenario with the maximum product as the “ideal” one, because it has both high probability and large energy saving. The “1” in Eq. (7)guarantees that we ﬁrst compare energy saving of scenarios; if energy saving are
equal, then we compare their probability. In this way, we choose a scenario thatmaximizes the total energy saving and its wake-up probability is higher than thethreshold of 0.5N , and solve the challenge in section 3.3.
At the last step, the current scenario is compared with the “ideal” one,and decisions of power mode switching are sent to the ubiquitous computing
environments for execution.
After learning the usage patterns from historical data, Gynapse repeats prediction and these four steps every time slice. In this way, it can handle powermode switching to maximize the total power saving while ensuring that responsetime does not exceed the required level.
5. Implementation and Evaluation
In this section, we shall discuss the details of implementation and evaluationof Gynapse. We ﬁrst describe the data and environment of our experiments, andthen discuss the preliminary results.
5.1 Experimental Environment
We use sensor data from MIT PlaceLab15) to evaluate our system. PlaceLab
is a 1000 sq. ft. apartment consisting of a living room, dining area, kitchen,small oﬃce, bedroom, full bath and half bath. 15 types of sensors are installedin PlaceLab: Interior conditions of the apartment are captured using distributed
temperature, humidity, bright, and barometric pressure sensors. The PlaceLabc 1977 Information Processing Society of Japan
⃝

1243

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction
Table 2 Devices used in Gynapse

Table 1 Sensors used in Gynapse
Type
Current
SwitchMITes OM
RFID
Motion
Water
Gas

Living
3
4
25
10
2
—
—

Dining
3
7
16
3
1
—
—

Kitchen
7
20
40
27
1
4
1

Oﬃce
2
3
14
13
1
—
—

Bedroom
2
5
15
12
1
—
—

Hallway
3
5
25
8
2
2
—

also features electrical current sensors, water ﬂow and gas ﬂow sensors. Small,
wired switches detect open/close events, such as the opening of linen closet. RFID
tags and MITes object movement sensors can be easily taped onto any non-wired
objects such as chairs, cups, remotes, tea-maker, and other objects people may
manipulate36) . A resident in the PlaceLab can wear up to three wireless 3-axis,
0-10 G accelerometers that measure limb motion. A wireless heart rate monitor(using a standard Polar chest strap) can also be worn. Five receivers spreadthroughout the apartment collect all wireless object motion, accelerometer, andheart rate data sent via the MITes wireless sensors36) . Nine infrared cameras, 9color cameras, and 18 microphones are distributed throughout the apartment incabinet components and above working surfaces. From these multi-modal sensors
a resident’s activities can be inferred15) .
People may argue that, in real life, no residents will install so many sensors intheir house because of cost and complexity. However, it is important for research
purpose to reveal the eﬀectiveness of energy saving for all sensors before we cannarrow them down. As a result, we select a set of 286 sensors that have close
correlation with device usage to build our system, which are shown in Table 1.
Since the sensor values are quite diﬀerent, Data Aggregator normalizes them⃗ =into a range between 0 and 1, and builds them into time series as Θ
)
( 1 2
M
⃗
⃗
⃗
⃗
is the observations of[O1 , O2 , . . . , OT ], where vector Ot = st , st , . . . , st
HHMM, as explained in section 4.
We use multi-modal sensors excluding current sensors to obtain the usage information of 28 electronic devices, which are listed in Table 2.
We use current sensors to obtain a single device’s wake-up time, and energy
saving (which is the diﬀerence of current readings between working and low power

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

LivingKitchen
DiningBedroom
HallwayOﬃce

devices
2 lights, TV, DVD player, speaker,
2 lights, microwave, coﬀee maker, tea maker, toasterdish washer, mixer, can opener, stove, garbage disposer
light, answer machinelight, lamp, alarm,light, washer, dryer
light, computer, monitor, fax machinemode). Figure 7 shows the current reading of living room light. Time at point
A and B are 19:06:29.693 and 19:06:34.287 respectively, so the wake-up time ofthis light is about 4.5 seconds. We can also ﬁnd the current diﬀerence between
working and low power mode is about 1500 mA. For multiple devices, we cancalculate their correlation of current status. Figure 8 shows the current status ofcoﬀee maker, toaster, and PC in a morning. We can observe the obvious linkage
between coﬀee maker and toaster. Statistically, we have the time series of current
consumed by coﬀee-maker and PC as CCM,t and CP C,t , so their normalized crosscorrelation can be calculated as:
)(
)
tn (
CCM,t − C̄CM CP C,t − C̄P C
1 ∑corr =
n − 1 t=tσCM σP C
1

where C̄ and σ are mean and standard deviation of time series, and n is the total
number of data points. During this period, the correlation between coﬀee makerand toaster is 0.7870, whereas the correlation between coﬀee maker and PC is-0.0968. The numerical results also prove our argument in section 3.1.
There are totally 23 days of data available, within which we use 9 days to train
our three-level HHMM, and the rest 14 days for evaluation.
5.2 Preliminary Results
In this paper, we have mainly discussed three components of Gynapse: 1) a
probabilistic model to learn residents’ usage patterns at device level, 2) a predictive mechanism to forecast the usage probability of devices, and 3) a control
framework to maximize the energy saving under the constraint of user required
response time. Correspondingly, we evaluate Gynapse from four prospects: 1)
the learning curve of probabilistic model, 2) the precision and recall rate of pre-c 1977 Information Processing Society of Japan⃝A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction

1000

500

A

0
19:06:20 19:06:27 19:06:34 19:06:41 19:06:48 19:06:55

Table 3 Precision and recall of predicted usage

Coffee Maker
20
100
7:00:00

7:15:00

7:30:00

7:45:00

8:00:00

8:15:00

8:00:00

8:15:00

Toaster
15000
10000
5000
0
7:00:00

7:15:00

7:30:00

7:45:00

PC
2000
1500
1000
500
7:00:00

Time

7:15:00

7:30:00

7:45:00

8:00:00

8:15:00

Time

Fig. 7 Current reading of light

Fig. 8 Usage correlation of devices

dictive mechanism, 3) the balance of energy saving and user required response
time, and 4) the total energy saving on each day.
5.2.1 Learning Curve
Figure 9 shows the learning curve of our three-level HHMM. At ﬁrst the likelihood is very low, because the training data is not enough and the parameters
have not been properly learned. As we add more training data, the likelihood improves gradually. After about 100 hours of training data, the likelihood becomesstable. Roughly speaking, we can consider the training data are now enough.
However, we have to be aware of two things. First, strictly speaking, the exact hours needed depends on the initial values of parameters, training policy,
and convergence criterion of the learning algorithm37) . E.g., if we require theconverge threshold to be 0.001, we obviously need more training data than athreshold of 0.1. Meek et al. provide more discussion about sample size andlearning curve in 38). Second, for some devices such as a stove, the resident does
not use it everyday. Therefore, in practice, we need more days to collect training
sample of stove than a heavily used device such as TV. The small peaks andvalleys on the curve after 100 hours reﬂect this point.
5.2.2 Precision and Recall
Table 3 shows the precision and recall of predicted device usage, which aredeﬁned as:P
P
Recall = T PT+F
P recision = T PT+F
P
N
where TP (True Positive) means we predict a device to be used, and it is actuallyused; FP (False Positive) means we predict a device to be used, but actually it is

IPSJ Journal

Vol. 18

Devices
Livinglight
TV
DVD player
Kitchenlight
microwave
coﬀee makertea maker
toasterdish washer
mixer
can opener
stove
garbage disposer

No. 4

1234–1249 (Apr. 1977)

Precision

Recall

95%
93%
89%

93%
92%
85%

96%
87%
85%
86%
83%
86%
82%
80%
81%
79%

93%
84%
83%
84%
82%
83%
81%
78%
80%
77%

Devices
Bedroom
light
lamp
alarm
Dininglight
answer machine
Hallwaylightwasher
dryer
Oﬃce
light
PC
fax machine

-40

1

-60
-80

likelihood

Current ( mA )B

Extra Energy Saving (kWh)1500

Current (mA) Current (mA) Current (mA)

1244

-100
-120
-140
-160
-180
0

30

60

90

120

150

180

210

Hours

Fig. 9 Learning curve

240

x 10

Precision

Recall

94%
91%
84%

93%
89%
81%

93%
86%

92%
82%

93%
89%
88%

91%
88%
85%

94%
92%
85%

93%
90%
82%

-3

Comb_50.5

Comb_30

Comb_2
Comb_0-0.5

Comb_4

Comb_1-1

Combination of Required Response Time

Fig. 10 Energy saving vs. Response time

not used; FN (False Negative) means we predict a device not to be used, but it
is actually used. Please note that, as explained in section 5.1, we determine theactual usage from multi-modal sensors excluding current sensor. In the example
of Fig.3, we use RFID tag on keyboard to determine the resident actually starts
or stops using PC, but not the current sensor of PC.
From Table 3, we can ﬁnd the predictive mechanism has better performancefor some devices such as lights, TV, and PC. This is because they are used more
frequently, so more training samples are available.
5.2.3 Energy Saving and Required Response TimeAs explained in section 4.3, control framework maximizes the energy saving
under the constraint of user required response time TRS . Since the upper limit ofc 1977 Information Processing Society of Japan
⃝

1245

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction
Table 4 Combination of required response time
Lights (sec)
4
2
4
4
4
4

TV (sec)
4
2
6
10
10
10

PC (sec)
4
2
6
10
30
44

TRS is a device’s wake-up time TW U , we use lights, TV, and PC to represent three
categories of devices according to their TW U : lights represent the devices with ashort wake-up time such as 4.5 seconds. Therefore, the resident should expect
the lights to response within 4 seconds⋆1 . On the other end, PC represents
the devices with a long wake-up time such as 44 seconds. Then the required
response time of PC can range from 0 to 44 seconds. TV represents the category
between these two. We set a benchmark (Comb 0), where the required responsetime is 4 seconds for all devices, and then calculate the extra energy saving of
diﬀerent combinations of required response time. Table 4 shows the diﬀerent
combinations.
Figure10 depicts the extra energy saving (compared with Comb 0) of diﬀerent
combinations. We can ﬁnd that, if the resident tolerates slow response fromdevices, such as Comb 5, more energy saving is achieved. In contrast, if theresident need immediate response from devices, such as Comb 1, less energy
saving is achieved. It clearly shows Gynpase’s capability of balancing energy
saving and required response time.
5.2.4 Total Energy Saving
After setting the user required response time, control framework calculates the“ideal” scenario, compares it with the current one, and switches power mode
accordingly. If a device is not used in “ideal” scenario, but its power status is in
working mode, then we can turn it into low power mode for saving energy. In
this way, we can calculate the total power consumption based on the decisions
from Gynapse.
Figure 11 shows the real and Gynapse-adjusted total power consumption of⋆1 For simplicity, we only use integer here.

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

2000

Real
Gynapse

1800

Power Consumption ( W )Combination
Comb 0Comb 1
Comb 2Comb 3
Comb 4
Comb 5

1600
1400
1200
1000
800
600
400
200
0
0:00

3:00

6:00

9:00

12:00

15:00

18:00

21:00

24:00

Time

Fig. 11 The real and Gynapse-adjusted power consumption of one day
Table 5 Power consumption and saving ratio of each day
DayReal (kWh)Gynapse (kWh)Saving Ratio
Day
Real (kWh)Gynapse (kWh)Saving Ratio

1
8.64
7.79
9.78%
8
9.22
8.58
7.01%

2
9.82
8.57
12.72%
9
14.35
12.37
13.77%

3
9.36
8.66
7.46%
10
10.63
9.00
15.32%

4
3.11
3.11
0.00%
11
9.10
7.83
14.03%

5
7.68
6.83
11.08%
12
9.09
7.90
13.01%

6
12.55
11.37
9.36%
13
10.51
9.26
11.89%

7
7.72
7.09
8.13%
14
9.59
8.47
11.77%

one day after the HHMM has been properly learned. We can ﬁnd that the energy
saving are mainly achieved during morning and evening, when multiple electronic
devices are used. While in the early morning and noon, since no devices are used,almost no power saving is achieved.
Table 5 shows the real power consumption, Gynapse-adjusted power consumption, and saving ratio on 14 evaluation days⋆2 . We can ﬁnd that Gynapse averagely saves about 11% of energy on these 14 days. The ﬂuctuation of the saving
ratio depends on the resident’s behavior. For instance, since the resident was⋆2 According to 39), the average household power consumption in Massachusetts is 21.17kWh
per day. Since PlaceLab is a 1000 sq. ft. apartment with a single person, it consumes lesselectricity than a typical 2000+ sq. ft. house with four persons.

c 1977 Information Processing Society of Japan
⃝

1246

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction

not at home on the 4th day, less power was consumed and no power saving wasachieved.
6. Conclusions and Discussions
In this paper, we have designed a ubiquitous power management system called
Gynapse, which uses multi-modal sensors to predict the exact usage of each device, and then switches their power modes based on predicted usage to maximize
the total energy saving under the constraint of user required response time. We
have discussed the challenges and solutions for three important components ofGynapse:• We have built a three-level Hierarchical Hidden Markov Model (HHMM) torepresent multiple residents and hierarchical structure of their activities, andlearn the adaptive device usage patterns in diﬀerent situations from multimodal sensors.
• Based on the learned HHMM, we have developed our predictive mechanism
in Dynamic Bayesian Network (DBN) scheme to precisely predict the usageof each device, with device’s wake-up time and user’s required response time
under consideration.
• With the predicted usage probability of each device, we have followed a fourstep process to balance the total energy saving and response time of devices
by switching their power modes according to the scenario with the highest
energy saving and probability.
Correspondingly, we have used PlaceLab data set to evaluate Gynapse fromfour prospects: 1) the learning curve of three-level HHMM, 2) the precisionand recall rate of predictive mechanism, 3) the balance of energy saving anduser required response time, and 4) the total energy saving on each day. The
preliminary results have demonstrated that as a ubiquitous power management
system, Gynapse has the capability to reduce power consumption while keeping
the response time not exceed user requirement. It provides a complementaryrather than competitive apporach to previous power mode switching systems.
6.1 Discussions
In this section, we provide some discussions about Gynapse as follows:
Undesirable aspects of overly energy saving: “Overly energy saving”

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

means a device should be in working mode, but it is switched oﬀ for savingenergy. As a result, it may cause two “undesirable aspects” as follows:• Incorrect switch-on/oﬀ: e.g., when the resident goes to restroom during his
working on PC, Gynapse incorrectly turns PC oﬀ. Although the resident isin a little break, Gynapse switches PC oﬀ incorrectly for saving energy.
• Delay of response: e.g., Gynapse does not turn on PC until the resident sits
in front of it, so he has to wait for a long wake-up time. In this case, theresident wants to use a device, and Gynapse correctly predicts that. However,for saving energy, Gynapse does not switch it on until the last moment.
In case 1, Gynapse incorrectly predicts the resident’s activity. Essentially,
it is impossible to completely eliminate the incorrect prediction. However, we
believe the possibility still exists to improve Gynapses accuracy. As we discussed
in section 4.2, a potential method is integrating user feedback with predictive
mechanism. Of course, the incorrect prediction cannot be completely avoided
even if we introduce user feedback into the system. When Gynapse incorrectlypredict device usage, the resident has to explicitly deﬁne his policies, like in arule-based system, or manually switch the devices. Incorrect prediction is thelimitation of Gynapse, and the choice among Gynapse, rule-based systems andmanual switching depends on the resident and his situation.
In case 2, Gynapse correctly predicts the resident’s activity, but the resident
may still feel frustration because of the long wake-up time. To solve this problem,
we provide a variable in predictive mechanism to reﬂect user’s required response
time. Then Gynapse tries to maximize the energy saving under the constraint ofuser’s required response time.
Comparison of Gynapse and automatic power-oﬀ mechanism interms of energy saving: We have already discussed in section 2 that the automatic power-oﬀ mechanism cannot handle complicated situations, such as theresident may watch TV when cooking in kitchen. However, in terms of energy
saving, it is still a simple and eﬀective method, which can be found in infraredcontrolled lights or an electric pot. A simple example is depicted in Fig. 12.
Assume the resident stops using an electric pot at T1 ; after a certain time TAF , it
is automatically switched oﬀ to low-power model (LP) at T2 . When the resident
starts using the pot again at T3 , it is switched on. We compare the energy savingc 1977 Information Processing Society of Japan
⃝

1247

A Ubiquitous Power Management System to BalanceEnergy Saving and Response Time based on Device-level Usage Prediction
Stop
Using
Automatic
power-off

TAF

Start
Using

WK
LP
TGF
WK

GynapseLP

T1

T2' T2

Time

T3

Fig. 12 Automatic power-oﬀ vs. Gynapse

of automatic power-oﬀ mechanism with Gynapse at switch-oﬀ (around T2 ) andswitch-on (around T3 ) respectively.
At switch-oﬀ, Gynapse uses the resident’s activity to help making decision. For
instance, if Gynapse ﬁnds the resident has left home after using the pot, it can′turn oﬀ the pot at T2 without waiting for a certain time. Whether Gynapse saves
more or less energy than automatic mechanism at switch-oﬀ actually depends on
the length of TAF and TGF : if TAF is longer, then Gynapse saves more energy;
if TAF is shorter, then Gynapse saves less energy.
At switch-on, since Gynapse predicts the resident’s activity to shorten responsetime of devices, it cannot save more energy than automatic power-oﬀ mechanism.
For instance, an electric pot may need some time to heat water to a certain
temperature, such as 100o C for tea. If Gynapse predicts that the resident isgoing to make some tea, it can turn on the pot to heat water before T3 . Hence,
the resident need not wait for a long response time. In contrast, the automatic
power-oﬀ mechanism will not start heating the water until the resident turns
it on. As a result, Gynapse can help the resident by shortening response time,
though this means less power saving.
Based on the discussion above, we can ﬁnd that for a simple device, whether
the automatic power-oﬀ mechanism may save more or less energy depends on thesummation of switch-oﬀ and switch-on parts. However, Gynapse can save energy
for a device used in complicated situations, which may not be easily handled by
the automatic power-oﬀ mechanism. In addition to energy saving, Gynapse also
considers the response time of devices.
Appropriate information for human activity prediction: Gynapse aims

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

at providing a ﬂexible scheme that can incorporate a variety of information into
the prediction of human activity, rather than comparing which information is
the most appropriate. As we explained in section 4, the input of our system is()
i⃗ t = s1t , s2t , . . . , sM
a time series of vectors O
t , where st corresponds to the valueof sensor i at time t. No matter this “sensor” is a real sensor or a virtual one,such as “a day of the week”, it can always be put into the vector. This scheme
provides great ﬂexibility to our choice of information. With the development ofresearches on sensors and human activity recognition, we can incorporate new
information into our system.
Comparison with other power management systems: We are fully awareof the importance of comparing Gynapse with previous power management systems. However, previous systems are all built in their own environments with
diﬀerent settings of devices and sensors. The lack of benchmark makes the comparison very diﬃcult if not impossible. Our evaluation with MIT PlaceLab data
set reﬂects our eﬀort to use a public data source, so it may be easier for otherresearchers to compare with us. However, since PlaceLab data is not designed forevaluating energy consumption, it inevitably has some limitations, and we have toleave some situations untested, such as multi-resident models and integration ofuser feedback. Currently, we are building a multi-modal sensor database mainlyfor power consumption evaluation in our laboratory. We believe the release of ourdata will provide a high quality database for further research and comparison.
References
1) Agency for Natural Resources and Energy: FY 2007 Energy Supply and Demand
Report (Final Report), (2009). (in Japanese).
2) Ministry of Internal Aﬀairs and Communications: Workshop Report concerning
ICT Policy for Correspondence to Global Warming Issue, (2008). (in Japanese).
3) Yamazaki, T., Jung, J., Kim, Y., Hahn, M., et al.: Energy Management in Home
Environment Using a Power Sensor Network, Technical Report IEICE , Vol. 107,
No.430, EE2007-56, pp.71-76, (2008). (in Japanese).
4) Kudo, H.: IT Applications for Energy Conservation J. IEICE , Vol.90, No.3, pp.
183 –190 (2007). (in Japanese).
5) Ishida, K., and Ito, Y.: Trend of Measurement and Control Technologies in IT
Era (4): Consumer Electronic Synchronization Control by HEMS, J. Society ofHeating, Air-Conditioning and Sanitary Engineers of Japan, Vol.80, No.5, pp.385-

c 1977 Information Processing Society of Japan
⃝

1248

A Ubiquitous Power Management System to BalanceEnergy Saving and Response Time based on Device-level Usage Prediction

393, (2006). (in Japanese).
6) Mozer, M. C.: The Neural Network House: An Environment that Adapts to its
Inhabitants, Proc. AAAI Spring Symp. on Intell. Environ. (M. Coen (ed.)), Menlo
Park, CA, AAAI Press, pp.110-114 (1998).
7) Mozer, M. C: Lessons from an adaptive house, Smart environments: Technologies,
Protocols, and Applications, J. Wiley & Sons, chapter12, pp.273-294 (2005).
8) Roy, A., Das Bhaumik, S.K., Bhattacharya, A., Basu, K., et al.: Location Aware
Resource Management in Smart Homes, PerCom2003 Proceedings, IEEE, pp.481–
488 (2003).
9) Harris, C., and Cahill, V. : Exploiting User Behaviour for Context-aware PowerManagement WiMob’2005 Proceedings, Vol.4, pp.22 – 130 (2005).
10) Harris, C., and Cahill, V. : An Empirical Study of the Potential for Context-Aware
Power Management UbiComp 2007 Proceedings, Vol.4717, Springer Berlin, pp.235–252 (2007).
11) Harle, R. K., and Hopper, A.: The Potential for Location-aware Power Management, UbiComp 2008 Proceedings , ACM, Vol.344, pp.302 – 311 (2008).
12) Fine, S., Singer, Y., and Tishby, N.: The Hierarchical Hidden Markov Model:
Analysis and Applications, Mach. Learn., Vol.32, pp.41–62 (1998).
13) Murphy, K., and Paskin, M.: Linear Time Inference in Hierarchical HMMs, Proc.
Neural Information Processing Systems, Canada (2001).
14) Murphy, K. P.: Dynamic Bayesian Networks: Representation, Inference and Learning, PhD thesis, UC Berkeley, Berkeley, (2002).
15) Intille, S. S., Larson, K., Tapia, E. M., Beaudin, J., et al.: Using a live-in laboratory for ubiquitous computing research, PERVASIVE 2006 Proceedings, Vol. LNCS
3968, Springer-Verlag, pp.349 – 365 (2006).
16) Lu, Y. H., De Micheli, G.: Comparing System-Level Power Management Policies,
IEEE Des. Test Comput., Vol.18, No.2, pp.10 – 19 (2001).
17) Stumpf, S., Sullivan, E., Fitzhenry, E., Oberst, I., et al.: Integrating Rich User
Feedback into Intelligent User Interfaces IUI ’08 Proceedings , ACM, (2008).
18) Culotta, A., Kristjansson, T., McCallum, A., and Viola, P.: Corrective Feedback
and Persistent Learning for Information Extraction, Artif. Intell., Vol.170, No.14,
pp.1101 – 1122 (2006).
19) Philipose, M., Fishkin, K. P., Perkowitz, M., Patterson, D. J., et al.: Inferring
Activities from Interactions with Objects, IEEE Pervasive Comput., Vol.3, No.4,
pp.50 – 57 (2004).
20) Yamahara, H., Takada, H., Shimakawa, H.: A Behavioral Pattern Adapted to
Individual for Providing Ubiquitous Service in Intelligent Space, Proc. 5th WSEAS
Intl. Conf. on Computational Intelligence, Man-Machine Systems and Cybernetics,
Venice, Italy, WSEAS, pp.84 – 92 (2006).
21) Moncrieﬀ, S., Venkatesh, S., West, G., and Greenhill, S.: Multi-modal Emotive
Computing in a Smart House Environment, Pervasive Mob. Comput., Elsevier,

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

Vol.3, No.2, pp.74 – 94 (2007).
22) Chetty, M., Brush, A. J. B., Meyers, B. R., and Johns, P.: It’s Not Easy Being
Green: Understanding Home Computer Power Management CHI 2009 Proceedings, ACM, pp.1033 – 1042 (2009).
23) Naeem, U., Bigham, J., and Wang, J.: Recognising Activities of Daily Life Using
Hierarchical Plans, EuroSSC 2007 Proceedings, Vol.4793, Springer Berlin, pp.175 –189 (2007).
24) Patterson, D.J., Fox, D., Kautz, H., and Philipose, M.: Fine-grained Activity
Recognition by Aggregating Abstract Object Usage Wearable Computers 2005 Proceedings, IEEE, pp.44 – 51 (2005).
25) National Institute of Population and Social Security Research: Summary of Household Projection for Japan, (2008). (in Japanese).
26) Theocharous, G.: Hierarchical Learning and Planning in Partially Observable
Markov Decision Processes , PhD thesis, Michigan State University, Lansing (2002).
27) Hoggan, E., Brewster, S. A., and Johnston, J.: Investigating the Eﬀectiveness of
Tactile Feedback for Mobile Touchscreens, CHI ’08 Proceedings, ACM, (2008).
28) Lee, J. H., and Spence, C.: Assessing the Beneﬁts of Multimodal Feedback on Dualtask Performance under Demanding Conditions, Proc. 22nd British HCI Group
Annual Conf. on HCI 2008 , Vol.1, (2008).
29) Martins, D. S., Biajiz, M., Prado, A. F., and Souza, W. L.: Implicit RelevanceFeedback for Context-aware Information Retrieval in UbiLearning Environments,
SAC ’09 Proceedings , ACM, (2009).
30) Kaufmann, O., Lorenz, A., Oppermann, R., Schneider, A., et al.: Implicit Interaction for Pro-active Assistance in a Context-adaptive Warehouse Application,
Mobility ’07 Proceedings , ACM, (2007).
31) Niculescu, R. S., Mitchell, T. M., Rao, R. B.: Bayesian Network Learning withParameter Constraints, J. Mach. Learn. Res., Vol.7, MIT Press, (2006).
32) Stumpf, S., Rajaram, V., Li, L., Wong, W. K., Burnett, M, et al.: Interacting
Meaningfully with Machine Learning Systems: Three Experiments, Technical Report 2007-46, EECS Oregon State University, (2007).
33) Zigoris, P., and Zhang, Y.: Bayesian Adaptive User Proﬁling with Explicit & Implicit Feedback, CIKM ’06 Proceedings , ACM, (2006).
34) Li, L., Muller, M. J., Geyer, W., Dugan, C., et al.: Predicting Individual Priorities of Shared Activities Using Support Vector Machines, Proc. 16th Conf. on
Information and Knowledge Management , ACM, pp.515–524 (2007).
35) Tanaka, K., Kishino, Y., Terada, T., and Nishio, S.: A Destination Prediction
Method Using Driving Contexts and Trajectory for Car Navigation Systems, SAC’09 Proceedings , ACM, pp.190–195 (2009).
36) Tapia, E. M., Intille, S. S., LOpez, L., and Larson, K.: The Design of a Portable
Kit of Wireless Sensors for Naturalistic Data Collection, PERVASIVE 2006 Proceedings, Springer-Verlag, (2006).

c 1977 Information Processing Society of Japan
⃝

1249

A Ubiquitous Power Management System to Balance Energy Saving and Response Time based on Device- level Usage Prediction37) Meek, C., Thiesson, B., and Heckerman, D.: The Learning Curve Method Applied
to Clustering, Microsoft Research Technical Report 2001-34, Morgan Kaufmann,(2001).
38) Meek, C., Thiesson, B., and Heckerman, D.: The Learning-Curve Sampling Method
Applied to Model-Based Clustering, J. Mach. Learn. Res., Vol.2, pp.397–418 (2002).
39) Energy Information Administration: U.S. Average Monthly Bill by Sector, Census
Division, and State 2007, (2009).

(Received July 8, 2009)(Accepted December 19, 2009)

Hua Si received his B.E. from Tsinghua University, Beijing,
China, in 2002, M.E. from the University of Tokyo, Tokyo, Japan,
in 2005, and M.F.E. from University of California, Berkeley,
United States, in 2008, respectively. He is currently a doctoral
student of the Research Center for Advanced Science and Technology at the University of Tokyo. His research interest is in thearea of ubiquitous computing and machine learning. He is a student member of IEEE, ACM, and IEICE.
Shunsuke Saruwatari received the Dr. Sci. degrees from the
University of Tokyo, Tokyo, Japan, in 2007. Since 2008, he had
been in the University of Tokyo and currently a research associateof the Research Center for Advanced Science and Technology at
the University of Tokyo. From 2007 to 2008, he stayed in University of Illinois at Urbana-Champaign as a visiting researcher.
His research interests are in the areas of wireless sensor networks,
real-time computing, real-time communication, and wireless networks. He is amember of IEEE, ACM, IEICE, and IPSJ.

IPSJ Journal

Vol. 18

No. 4

1234–1249 (Apr. 1977)

Masateru Minami received his B.E. and M.E. degrees from
Shibaura Institute of Technology, Tokyo, Japan, in 1996 and 1998,
respectively, and the Ph.D. degree from the University of Tokyoin 2001. He is currently an Associate Professor of the ResearchCenter for Advanced Science and Tech- nology at the Universityof Tokyo. From 2003 to 2007 he was an Assistant Professor of the
Department of Elec- tronic Engineering at Shibaura Institute ofTechnology. From 2005 to 2006, he stayed in the Ecole Polytechnique Federalede Lausanne (EPFL) as a visiting researcher. His research interest is in the areaof ubiquitous com- puting and sensor networks. He is a member of the IEEE,
ION GPS, IEICE and IPSJ.
Hiroyuki Morikawa received the B.E. M.E. and Dr. Eng. degrees in electrical engineering from the University of Tokyo, Tokyo,
Japan, in 1987, 1989, and 1992, respectively. Since 1992, he had
been in the University of Tokyo and currently a full professor of
the Research Center for Advanced Science and Technology at the
University of Tokyo. From 1997 to 1998, he stayed in Columbia
University as a visiting research associate. From 2002 to 2006, he
was a group leader of NICT Mobile Networking Group. His research interests
are in the areas of computer networks, ubiquitous networks, mobile computing,wireless networks, photonic Internet, and network services. He served as a technical program committee chair of many IEEE/ACM conferences and workshops,
Director of IEICE, Editor-in-Chief of IEICE Transactions of Communications,and sits on numerous telecommunications advisory committees and frequently
serves as a consultant to government and companies. He received more than 20
awards including the IPSJ best paper award in 2006, IEICE best paper award in
2002 and 2004, Info-Communications Promotion Month Council President Prizein 2008, DoCoMo Mobile Science Award in 2009. He is a fellow of IEICE, and amember of IEEE, ACM, ISOC, IPSJ, and ITE.

c 1977 Information Processing Society of Japan⃝