ADAPTIVE POWER MANAGEMENT OF
ENERGY HARVESTING SENSOR NODESUSING REINFORCEMENT LEARNING
A Comparison of Q-Learning and SARSA Algorithms
適応的電力制御を行う環境発電駆動センサノードの強化学習戦略の比較評価

SWoPP 2017
SHASWOT SHRESTHAMALIMASAAKI KONDOHIROSHI NAKAMURA
THE UNIVERSITY OF TOKYOINTRODUCTION
 Use Reinforcement Learning (RL) for power

management in Energy Harvesting Sensor Nodes(EHSN)
 Adaptive control behavior
 Near-optimal performance Comparison between different RL algorithms
 Q-Learning SARSA25-Mar-202

ENERGY HARVESTING SENSOR NODE CONCEPTHarvested
EnergyRF TransceiverPowerManager

MCUMemorySensor

Mixed SignalCircuits• CONSTRAINTS• Sensor node has to be operating atALL times• Battery cannot be completelydepleted• Battery cannot be overcharged(exceed 100%)• Battery size is finite• Charging/discharging rates arefinite

25-Mar-20

3

OBJECTIVE: NODE-LEVEL ENERGY NEUTRALITY• We want to use ALL the energy that isharvested.
• One way of achieving that is by ensuring

EnergyHarvested

EnergyConsumed

node level energy
neutrality – the condition when
the amount of energy harvested equals
the amount of energy consumed.

• Autonomous Perpetual
operation can be achieved
25-Mar-20

4

CHALLENGES

Environmental Sensor Networks – P.I. Corke et. al.

MOVING SENSORS

https://sites.google.com/site/sarmavrudhula/home/research/energy-management-of-wireless-sensornetworks

DIFFERENT ENVIRONMENTS

DIFFERENT SENSORS
http://www.mdpi.com/sensors/sensors-12-02175/article_deploy/html/images/sensors-1202175f5-1024.png

25-Mar-20

5

SOLUTION
Preparing heuristic, user-defined contingency solutions
for all possible scenarios is impractical.

We want a one-size-fits-all solution
sensor nodes that are capable of:• autonomously learning optimal
strategies• adapting once they have been
deployed in the environment.

25-Mar-20

6

SOLUTION➢ Use RL for
adaptive control➢ Use a solar
energyharvesting
sensor node as acase example
25-Mar-20

7

Q-Learning Results (ETNET 2017)100%

Higher Efficiency

90%

𝐴𝑐𝑡𝑢𝑎𝑙 𝐷𝑢𝑡𝑦 𝐶𝑦𝑐𝑙𝑒
𝐴𝑐ℎ𝑖𝑒𝑣𝑎𝑏𝑙𝑒 𝑀𝑎𝑥𝑖𝑚𝑢𝑚 𝐷𝑢𝑡𝑦 𝐶𝑦𝑐𝑙𝑒

80%
70%
60%
50%

Efficiency(%)

40%

Energy Wasted(%)

30%
20%

Lower Waste

10%0%

Naïve

Duty Cycle is
proportional tobattery level

Kansal

Our methodusing RL

Fix duty cycle forpresent day by
predicting totalenergy for next day

ETNET 2017 (Kumejima)

𝑇𝑜𝑡𝑎𝑙 𝐸𝑛𝑒𝑟𝑔𝑦 𝑊𝑎𝑠𝑡𝑒𝑑
𝑇𝑜𝑡𝑎𝑙 𝐸𝑛𝑒𝑟𝑔𝑦 𝐻𝑎𝑟𝑣𝑒𝑠𝑡𝑒𝑑

𝐸𝑛𝑒𝑟𝑔𝑦 𝑊𝑎𝑠𝑡𝑒 = 𝐸𝑛𝑒𝑟𝑔𝑦 𝐻𝑎𝑟𝑣𝑒𝑠𝑡𝑒𝑑−𝑁𝑜𝑑𝑒 𝐸𝑛𝑒𝑟𝑔𝑦− 𝐶ℎ𝑎𝑟𝑔𝑖𝑛𝑔 𝐸𝑛𝑒𝑟𝑔𝑦

8Q-Learning (ETNET 2017)❑ Demonstrated that RL approaches outperform
traditional methods.
❑ Limitations•State explosion• 200 x 5 x 6 states• Q-table becomes too large to train using random policy• Long training times• Required 10 years worth of training• Reward function did not reflect the true objective ofenergy neutrality.
ETNET 2017 (Kumejima)

9REINFORCEMENT LEARNING
IN A NUTSHELL

25-Mar-20

10

REINFORCEMENT LEARNING
What action
should I take toaccumulate
total maximumreward?

• Type of Machine Learning based on experience rather than instruction• Map situations (states) into actions – and receive as much reward aspossible
OBSERVATIONS:

Battery Level
Energy Harvested
Weather Forecast

REWARD, New State

Agent(Power Manager)
25-Mar-20

ACTION: Choose Duty Cycle

Environment
11

REINFORCEMENT LEARNING• IMPORTANT CONCEPTS▫ Q-VALUE▫ ELIGIBILITY TRACES

25-Mar-20

12Q-VALUE
r1• To give a measure of the“goodness” of an action
in a particular state, weassign each state-action
pair a Q-value:
Q(state, action)• Learned from past(training) experiences.
• Higher Q-value → better the choice ofaction for that state.
• Q(s,a) value is the expected cumulative
reward that you can get starting fromstate s and taking action a25-Mar-20

State

sj
a1
𝑄(𝑠𝑖 , 𝑎1)r2

a2StateStatesi𝑄(𝑠𝑖 , 𝑎2)a3

𝑄(𝑠𝑖 , 𝑎3)skr3Statesl13

Q-VALUE• To give a measure of the“goodness” of an action
in a particular state, weassign each state-action
pair a Q-value:
Q(state, action)• Learned from past(training) experiences.
• Higher Q-value → better the choice ofaction for that state.
• Q(s,a) value is the expected cumulative
reward that you can get starting fromstate s and taking action a25-Mar-20

State

sjr2
State

siStatea2

𝑄(𝑠𝑖 , 𝑎2)skStatesl

14

LEARNING Q-VALUESTO FIND 𝑸 𝒔𝒌 , 𝒂𝒌
• Start with arbitrary guesses for 𝑄 𝑠𝑘 , 𝑎𝑘
• Update 𝑄 𝑠𝑘 , 𝑎𝑘 incrementally towards the target value (Bootstrapping)• General Update Rule

𝑁𝑒𝑤𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒 ← 𝑂𝑙𝑑𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒 + 𝑆𝑡𝑒𝑝𝑆𝑖𝑧𝑒[𝑇𝑎𝑟𝑔𝑒𝑡 − 𝑂𝑙𝑑𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒]
𝑁𝑒𝑤𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒 ← 1 − 𝑆𝑡𝑒𝑝𝑆𝑖𝑧𝑒 × 𝑂𝑙𝑑𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒 + 𝑆𝑡𝑒𝑝𝑆𝑖𝑧𝑒 × 𝑇𝑎𝑟𝑔𝑒𝑡
𝑄 𝑠𝑘 , 𝑎𝑘 ← 1 − 𝛼 𝑄 𝑠𝑘 , 𝑎𝑘 + 𝛼 × 𝑇𝑎𝑟𝑔𝑒𝑡?
25-Mar-20

15

SARSA VS Q-LEARNING• Agent starts at state sk and takes some action ak according to policy .
• Receives a reward rk and is transported to new state sk+1.

SARSA

Q-LEARNING• The agent considers taking thenext action ak+1.
• The Q-value Q(sk,ak) is thenupdated.

• The agent assumes the next action willbe the action with the highest Q-value.
• The Q-value Q(sk,ak) is then updated.

𝑄𝜋 𝑠𝑘 , 𝑎𝑘 ← 1 − 𝛼 𝑄𝜋 𝑠𝑘 , 𝑎𝑘 +α[𝑟𝑘 + 𝛾𝑄𝜋 𝑠𝑘+1 , 𝑎𝑘+1 ]𝑄 𝑠𝑘 , 𝑎𝑘 ← 1 − 𝛼 𝑄 𝑠𝑘 , 𝑎𝑘 +α[𝑟𝑘 + 𝛾 max 𝑄(𝑠𝑘+1 , 𝑎)]𝑎• -greedy policy is used i.e. random actions are taken with probability  to allow
exploration.
25-Mar-20

16

SARSA VS Q-LEARNINGSARSA

𝑄𝜋 𝑠𝑘 , 𝑎𝑘 ← 1 − 𝛼 𝑄𝜋 𝑠𝑘 , 𝑎𝑘 + α[𝑟𝑘 + 𝛾𝑄𝜋 𝑠𝑘+1 , 𝑎𝑘+1 ]𝑁𝑒𝑤𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒 ← 1 − 𝑆𝑡𝑒𝑝𝑆𝑖𝑧𝑒 × 𝑂𝑙𝑑𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒 + 𝑆𝑡𝑒𝑝𝑆𝑖𝑧𝑒 × 𝑇𝑎𝑟𝑔𝑒𝑡

Q-Learning

25-Mar-20

𝑄 𝑠𝑘 , 𝑎𝑘 ← 1 − 𝛼 𝑄 𝑠𝑘 , 𝑎𝑘 + α[𝑟𝑘 + 𝛾 max 𝑄(𝑠𝑘+1 , 𝑎)]𝑎17

SARSA VS Q-LEARNING
SARSA

Q-Learning On-policy learning: Off-policy learning:

updates the policy it is using during
training

 final learned policy is the same

regardless of training methods

 Update is carried out by considering Assumes the best actions will always

the next action to be taken

be taken Faster convergence but requires an Takes longer to converge

initial policy.

 Difficult to integrate with linear Easier to integrate with function

function approximation

approximation

25-Mar-20

SARSA

Q-Learning

Choosing NextAction

-greedy policy

-greedy policy

Updating Q

-greedy policy

Greedy policy

18

ELIGIBILITY TRACES• In our model, one action is taken every hour. The reward is awarded at the end of 24hours. A single action cannot justify the reward at the end. A series of 24 state-action
pairs are responsible for the reward.
• To update the Q-values of the appropriate state-action pairs, we introduce a memory
variable, 𝑒(𝑠, 𝑎), called the eligibility trace.
• 𝑒 𝑠, 𝑎 for ALL state-action pairs decays by 𝜆 at every time step.
• If the state-action pair 𝑠𝑘 , 𝑎𝑘 is visited, 𝑒 𝑠𝑘 , 𝑎𝑘 is incremented by one.
State 1

Action 1

State 2

Action 2…

State 24

Action 24

REWARD

Update Q(State 24, Action 24)
Update Q(State 2, Action 2)

25-Mar-20

Update Q(State 1, Action 1)

21

SARSA() AND Q-LEARNING ()• SARSA() – integrate eligibility traces with SARSA algorithm• Q() – integrate eligibility traces with Q-Learning algorithm• , 0 < 𝜆 < 1, is the strength with which Q-values of early
contributing state-action pairs are updated as a consequenceof the final reward.

25-Mar-20

22

ADAPTIVE POWER CONTROL USING
REINFORCEMENT LEARNING ALGORITHMS• SARSA() – SARSA with eligibility traces• SARSA• Q() – Q-Learning with eligibility traces• Q-Learning

25-Mar-20

23

STATE DEFINITION
State at 𝑡𝑘 = 𝑆𝑑𝑖𝑠𝑡 𝑘 , 𝑆𝑏𝑎𝑡𝑡 𝑡𝑘 , 𝑆𝑒ℎ𝑎𝑟𝑣𝑒𝑠𝑡 𝑡𝑘 , 𝑆𝑑𝑎𝑦 𝑡𝑘
Distance fromBattery,energy neutrality, 𝑆𝑏𝑎𝑡𝑡 (𝑡𝑘 )𝑆𝑑𝑖𝑠𝑡 (𝑡𝑘 )Harvested Energy,𝑆𝑒ℎ𝑎𝑟𝑣𝑒𝑠𝑡 (𝑡𝑘 )Weather Forecast,
𝑆𝑑𝑎𝑦 (𝑡𝑘 )- 20000 mWh

Low (< 20%)0 mWh

- 19000 mWh

Mid (20% to 80%)

0 to 100 mWh

High (> 80%)

100 mWh to 500 mWh

Partly Cloudy

0 mWh

500 mWh to 1000 mWhFair⋮

1000 mWh to 1500 mWh

Sunny19000 mWh

1500 mWh to 2000 mWh

Very Sunny

20000 mWh

> 2000 mWh⋮25-Mar-20

Very little sun
Overcast24ACTION SPACE• Choose duty cycle of the sensor node𝐴 = 𝑎 𝑡𝑘 ∈ 1,2,3,4,5
ACTION DUTY CYCLE(%)
𝑎 𝑡
122040ENERGYCONSUMED PER
HOUR (mWh)
1002003456080100300
400500

𝑘25-Mar-2025

REWARD FUNCTION• Awarded at the end of an episode (day).
• Each episode consists of 24 one-hour epochs.
• We want the net energy difference between
initial and final battery levels to be zero.
• Use a reward scheme that depends on Energy
Neutral Performance (ENP) at the end of theepisode (𝑡𝑘 = 𝑇).
• Energy Neutral Performance can be defined hereas▫ |𝐼𝑛𝑖𝑡𝑖𝑎𝑙 𝑏𝑎𝑡𝑡𝑒𝑟𝑦 𝑙𝑒𝑣𝑒𝑙 – 𝐹𝑖𝑛𝑎𝑙 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑏𝑎𝑡𝑡𝑒𝑟𝑦 𝑙𝑒𝑣𝑒𝑙|

25-Mar-20

26

TRAINING AND TESTING• Training:
Tokyo,Year 2010

• Testing:Tokyo,Year 2010/2011
Wakkanai, Year 2010/2011• Wakkanai has a much colder climate than that ofTokyo and received much lesser solar radiation.
• We observe the adaptive behavior of our solution
when the location of implementation is different
from the location of its training
25-Mar-20

27

RESULTS

25-Mar-20

29

SARSA VS Q-LEARNING

Battery Percentage

Battery overflows and has
to be reset to initial level(60%)

SARSA() battery
profile does not need
to be reset

SARSA() (with eligibility traces)
SARSA
Q() (with eligibility traces)
Q-Learning

Day (2011,Tokyo)

25-Mar-2030

ENERGY NEUTRAL OPERATION
Battery profiles for SARSA andOffline Policy are very similar• SARSA(λ)compared withOptimal Policy

Tokyo 2011

• Optimal Policy▫ Theoretical upper limit▫ Calculated using future
information and linear
programming techniques

Duty Cycle (%)

10080
604020Battery (%)

8060
40
20
025-Mar-20

Optimal Policy using
non-causal data

ProposedMethod

Our method comesvery close to the
optimal solution

Solar Energy ProfileDAY 294812Time (Hour)

16202431SARSA VS Q-LEARNING• Every day thebattery is reset toinitial battery level

𝐸𝑁𝑃 = 𝐵𝑎𝑡𝑡𝑒𝑟𝑦 𝑎𝑡 00: 00 − 𝐵𝑎𝑡𝑡𝑒𝑟𝑦 𝑎𝑡 23: 59
𝐸𝑁𝑃 = |60% 𝑜𝑓 𝐵𝑀𝐴𝑋 − 𝐵𝑎𝑡𝑡𝑒𝑟𝑦 𝑎𝑡 23: 59|• ENP (as a percentageof maximum battery
capacity, BMAX) isobserved at the endof each day of theyear.

25-Mar-20

32

SARSA VS Q-LEARNING

25-Mar-20

33

OBSERVATIONS• SARSA(λ) – BEST PERFORMANCE.
• Q(λ) – WORST PERFOMANCE.
• The “high” learning rate causes Q-values to oscillate with large
amplitudes and the policy cannot converge.
• A lower learning rate shows better performance but at expense of longer
learning times.
• SARSA methods have a generally robust performance as compared to QLearning.
• Using eligibility traces with SARSA enhances the performance.

25-Mar-20

35

SUMMARY Adaptive Control is achieved by using SARSA RL methods .
 Results from SARSA RL are near optimal.

 SARSA() outperforms Q-Learning methods.

25-Mar-20

36

For further details about our work using SARSA(λ), please refer to our paper to bepresented in EMSOFT 2017 and published in ACM TECS Journal.

Adaptive Power Management in Solar Energy Harvesting Node using
Reinforcement Learning

THANK YOU FOR LISTENING
ANY COMMENTS OR QUESTIONS ARE WELCOME

shaswot@hal.ipc.i.u-tokyo.ac.jp
This work was partially supported by JSPS KAKENHI Grant Number 16K12405.

25-Mar-20

37