Adaptive Power Management inSolar Energy Harvesting WirelessSensor Node using
Reinforcement Learning
SHASWOT SHRESTHAMALIMASAAKI KONDOHIROSHI NAKAMURA
THE UNIVERSITY OF TOKYOEMBEDDED SYSTEMS WEEK, EMSOFT 2017, SEOUL

25-Mar-20

INTRODUCTION• Energy Harvesting Wireless Sensor

 An energy buffer (battery) Energy harvesting module(s) (e.g. solarpanels)• IoT will require (trillions of) diverse

sensor nodes deployed in different
environments.

RFTransceiverPowerManagerμ-controller

MemorySensor

Mixed SignalCircuits• Sensor Nodes should workautonomously and perpetually.

EMSOFT 2017, SEOUL

Nodes (EHWSNs) are wirelesssensor nodes withHarvested
EnergyBlock diagram of an EHWSN Maximize utility of sensor node Sustainable and maintenance-free2

25-Mar-20

PROBLEM DEFINITION

ENERGY HARVESTED = ENERGY CONSUMED

Node Level Energy Neutrality

EMSOFT 2017, SEOUL

Perpetual operation and maximization of sensor nodeutility can be achieved if:3

•25-Mar-20

THE PROBLEM
Unreliable energy harvesting•Strategies change with changes in environment• Change in location• Change in climate• Change in device parameters•

EMSOFT 2017, SEOUL• Unpredictable energy profiles• Predictions are unreliable

Scaling• Billions/trillions diverse sensors deployed in unique workingenvironments

4ResearchApproach

Limitations

Power management in energy
harvesting sensor networks,
Kansal et. al (2007)

Predict energy to beharvested and determine
duty cycle

Performance dependent on
prediction mechanism

Adaptive control of duty
cycling in energy-harvesting
wireless sensor networks,
Vigorito et. al (2007)

Linear Quadratic Control
SystemHyper parameters need tobe manually adjusted

A learning theoretic approachto energy harvesting
communication system
optimization, Blasco et. al(2013)

Reinforcement Learning

Applicable for sensor nodes
with communications as the
only power consuming
operation.

25-Mar-20

EMSOFT 2017, SEOULPREVIOUS APPROACHESTO ACHIEVINGNODE LEVEL ENERGY NEUTRALITY

525-Mar-20

SOLUTION

We want a one-size-fits-all solution
i.e. sensor nodes that:• learns the optimal strategy through▫ Context aware action – perception –learning cycle

EMSOFT 2017, SEOUL

Hand-engineered solutions for all
possible scenarios is impractical.

• adapts once they have beendeployed in the environment.

625-Mar-20

PROBLEM DEFINITION• ENERGY HARVESTED = ENERGY CONSUMED

 Node Level Energy Neutrality• Battery is never completely full or depleted• Sensor node maintains a minimum level of operation at alltimes
 Duty Cycling

EMSOFT 2017, SEOUL

Perpetual operation and maximization of sensor nodeutility can be achieved if:

7

25-Mar-20

SYSTEM MODEL a load that consumes
power depending on
its duty cycle
 Higher power
consumption implies
higher utility sensing/communication
functions areirrelevant.

EMSOFT 2017, SEOUL• Solar EHWSN• Use Reinforcement

Learning (RL) to arriveat an optimal control
policy.

825-Mar-20

REINFORCEMENT LEARNING• Type of machine learning based on experience rather thaninstructions• Agent interacts with environment to receive rewards.

GOAL: Maximize the total (discounted) CUMULATIVE
reward.

EMSOFT 2017, SEOUL Evaluative feedback instead of Instructive feedback

925-Mar-20

REINFORCEMENT LEARNING• Type of machine learning based on experience rather thaninstructions• Agent interacts with environment to receive rewards.

OBSERVATIONGOAL: Maximize the total (discounted) CUMULATIVE
reward.

POWER
MANAGER(AGENT)ENVIRONMENT(Battery, Weather,Solar Output)

EMSOFT 2017, SEOUL Evaluative feedback instead of Instructive feedback

1025-Mar-20

REINFORCEMENT LEARNING• Type of machine learning based on experience rather thaninstructions• Agent interacts with environment to receive rewards.

GOAL: Maximize the total (discounted) CUMULATIVE
reward.
50% Battery
20% Solar Output

State = s

POWER
MANAGER(AGENT)

ENVIRONMENT(Battery, Weather,Solar Output)

EMSOFT 2017, SEOUL Evaluative feedback instead of Instructive feedback1125-Mar-20

REINFORCEMENT LEARNING• Type of machine learning based on experience rather thaninstructions• Agent interacts with environment to receive rewards.

GOAL: Maximize the total (discounted) CUMULATIVEreward.
50% Battery
20% Solar Output

State = s

40%

POWER
MANAGER

60%(AGENT)

ENVIRONMENT(Battery, Weather,Solar Output)

EMSOFT 2017, SEOUL Evaluative feedback instead of Instructive feedback

80%

Possible Actions:
Choose a duty cycle

1225-Mar-20

REINFORCEMENT LEARNING• Type of machine learning based on experience rather thaninstructions• Agent interacts with environment to receive rewards.

GOAL: Maximize the total (discounted) CUMULATIVE
reward.
50% Battery
20% Solar Output

State = s

POWER
MANAGER(AGENT)

Duty Cycle = 40%

Action = a

ENVIRONMENT(Battery, Weather,Solar Output)

EMSOFT 2017, SEOUL Evaluative feedback instead of Instructive feedback

13

REINFORCEMENT LEARNING• Type of machine learning based on experience rather thaninstructions• Agent interacts with environment to receive rewards.

GOAL: Maximize the total (discounted) CUMULATIVE
reward.
50% Battery
20% Solar Output

State = s

POWER
MANAGER

Duty Cycle = 40%

Action = a(AGENT)

Reward = 530% Battery
40% Solar Output

Reward = r, New State = s’

ENVIRONMENT(Battery, Weather,Solar Output)

EMSOFT 2017, SEOUL Evaluative feedback instead of Instructive feedback

14

25-Mar-20

STATE DEFINITION
State at epoch 𝑡𝑘 = 𝑆𝑑𝑖𝑠𝑡 𝑡𝑘 , 𝑆𝑏𝑎𝑡𝑡 𝑡𝑘 , 𝑆𝑒ℎ𝑎𝑟𝑣𝑒𝑠𝑡 𝑡𝑘 , 𝑆𝑑𝑎𝑦 𝑡𝑘
Battery,
𝑆𝑏𝑎𝑡𝑡 (𝑡𝑘 )Harvested Energy,𝑆𝑒ℎ𝑎𝑟𝑣𝑒𝑠𝑡 (𝑡𝑘 )- 20000 mWh

Low (< 20%)0 mWh

- 19000 mWh

Mid (20% to 80%)

0 - 100 mWh

High (> 80%)

100 mWh - 500 mWh

Partly Cloudy

0 mWh

500 mWh - 1000 mWhFair⋮

1000 mWh - 1500 mWh

Sunny19000 mWh

1500 mWh - 2000 mWh

Very Sunny

20000 mWh

> 2000 mWh⋮

Weather Forecast,𝑆𝑑𝑎𝑦 (𝑡𝑘 )Very little sunOvercast

EMSOFT 2017, SEOUL

Distance fromenergy
neutrality,𝑆𝑑𝑖𝑠𝑡 (𝑡𝑘 )15

25-Mar-20

ACTION SPACE

𝐴 = 𝑎 𝑡𝑘 ∈ 1,2,3,4,5
ACTIONDUTY CYCLE(%)

ENERGY CONSUMEDPER HOUR (mWh)

120100234060200300

4580100400500

𝑎 𝑡𝑘

EMSOFT 2017, SEOUL

Choose duty cycle of the sensor node16

Battery Level (3)Weather Forecast (6)Harvested Energy (7)Energy Neutral Performance (ENP) (41)• Current battery – Optimal battery level
State = sPOWER
MANAGER(AGENT)

Calculated using statistical
data about the energy
harvesting environment

Discrete Duty Cycles(20%, 40%, 60%, 80%, 100%)
Action = a
An action is executed every hour
1 hour = 1 EPOCH
24 epochs = 1 EPISODEENVIRONMENT(Battery, Weather,Solar Output)

EMSOFT 2017, SEOUL••••

25-Mar-20

REINFORCEMENT LEARNING• SINGLE scalar value• Rewarded at the end of a day (episode)• Based on deviation of battery fromoptimal value
Reward = r

17

25-Mar-20• Awarded at the end of an episode (day).
• Ideally, difference between initial and final battery levels = 0• Reward scheme depends on Terminal Energy Neutral Performance(TENP) i.e. ENP at the end of the episode.
▫ Terminal Energy Neutral Performance is defined here as|𝐼𝑛𝑖𝑡𝑖𝑎𝑙 𝑏𝑎𝑡𝑡𝑒𝑟𝑦 𝑙𝑒𝑣𝑒𝑙 – 𝐹𝑖𝑛𝑎𝑙 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑏𝑎𝑡𝑡𝑒𝑟𝑦 𝑙𝑒𝑣𝑒𝑙|

EMSOFT 2017, SEOUL

RewardFunction18

25-Mar-20

THE LEARNING PROCESS• Simulate using historical weather data for Tokyo, 2010.

and remembers them.

• Learning Algorithm: SARSA() Learning• Compare with Offline Policy for 2011

 Offline Policy is calculated using assuming an omniscient solar
energy predictor and Linear Programming methods.
 Gives the optimal policy.
 This is not a realistic solution as it requires perfect information
about the future.

EMSOFT 2017, SEOUL• Agent tries various strategies, learns which policies are best

 Only for comparison purposes1925-Mar-20

LEARNING
= 0.1
= 0.1
= 0.8
= 0.8
= 104

EMSOFT 2017, SEOULTRAINING DAYS

𝛼 learning rate
𝜖 exploration ratio
𝛾 discount factor
𝜆 trace−decay parameter
𝑁 number of iterations

20

25-Mar-20

SARSA(λ)
Learning
DAY 53,2010
Tokyo

EMSOFT 2017, SEOULLEARNING

21

25-Mar-20

REWARDSARSA(λ)
Learning
DAY 53,2010
Tokyo
ITERATIONSEMSOFT 2017, SEOULLEARNING

22

25-Mar-20

Q-Values
Epoch 2
S32
a=2

S57

a=4

S1

Epoch 3

Epoch 1argmax 𝑄(𝑠, 𝑎)

S809

a=5

REWARD465

Epoch 24𝑎

𝑄(𝑠, 𝑎)

-2.5

483.7

…

(𝑠, 𝑎)(1,1) (1,2) (1,3) (1,4) (1,5) … (32,4) … (57,2) … (809,5)

…

-1.7

0.5

-5.8

0.1

…

30.7

…

121.3

…

Each state-action pair (𝑠, 𝑎) is associated with a Q-value 𝑄(𝑠, 𝑎)
for a particular policy 𝜋.
𝑄(𝑠, 𝑎) is the expected cumulative reward if you take action 𝑎 atstate 𝑠 and follow 𝜋.

EMSOFT 2017, SEOULa=323

➢ Each state action pair is initialized to an eligibility value (trace), 𝑒 𝑠, 𝑎 = 0• Every time (𝑠, 𝑎) is visited, 𝑒 𝑠, 𝑎 = 𝑒 𝑠, 𝑎 + 1• Otherwise, 𝑒 𝑠, 𝑎 decays by a factor of 𝛾𝜆.
• The value of 𝑒(𝑠, 𝑎) determines how influential that state-action pair
was in obtaining the reward at the end of an episode.
➢ Agent starts at state 𝑠𝑘 and takes some action 𝑎𝑘 according to policy .
➢ It receives a reward 𝑟𝑘 and is transported to a new state 𝑠𝑘+1 .
➢ The agent considers taking the next action 𝑎𝑘+1 .
➢ The Q-value 𝑄 𝜋 (𝑠𝑘, 𝑎𝑘) is then updated as:
𝑄 𝜋 𝑠𝑘 , 𝑎𝑘 ← 𝑄 𝜋 𝑠𝑘 , 𝑎𝑘 + α𝑒 𝑠, 𝑎 [𝑟𝑘 + 𝛾𝑄 𝜋 𝑠𝑘+1 , 𝑎𝑘+1 − 𝑄 𝜋 𝑠𝑘 , 𝑎𝑘 ]25-Mar-20

EMSOFT 2017, SEOULSARSA()• -greedy policy is used i.e. random actions are taken with probability to allow exploration. Otherwise greedy actions are executed.

24• Wakkanai• Much colder climate• Average Annual Temp =6.2°C• Observe behavior at alocation that has neverbeen experienced

25-Mar-20• Tokyo• Training grounds• Average Annual Temp =15.6°CEMSOFT 2017, SEOULIMPLEMENTATION

25

25-Mar-20

RESULTS

Battery profiles
for SARSA andOptimal Policy
are very similar

Tokyo, 2011

Near Perfect
Node Level
Energy Neutrality

EMSOFT 2017, SEOUL• Comparison with omniscient Offline Policy• Near Perfect Energy Neutral Performance26

25-Mar-20

RESULTS• Implemented inTokyo, 2011
• Adaptation tochange inweather

EMSOFT 2017, SEOUL• Trained inTokyo, 201027

25-Mar-20

RESULTSSARSA Policy and Offline Policy have similar battery profiles

EMSOFT 2017, SEOUL• Trained with Tokyo 2010 weather• Implemented in Wakkanai, 201128

25-Mar-20

RESULTSEMSOFT 2017, SEOUL

Battery Percentage (%)• Trained in Tokyo, 2010
• Implemented in Wakkanai, 2011
• Weather Forecast enhances perfomance29

25-Mar-20

RESULTSTokyo
31-Oct-2011

Watermarked, dashedlines are corresponding
values for full solar
panel capacity

EMSOFT 2017, SEOUL• Half Solar Panel Capacity• After training for 1000 iterations with 𝛼 = 0.1 and 𝜖 = 0.730

25-Mar-20

RESULTSTokyo31-Oct-2011

Watermarked, dashedlines are corresponding
values for full solar
panel capacity

EMSOFT 2017, SEOUL• Node Power Consumption increases by 2.5 times• After training for 1000 iterations with 𝛼 = 0.1 and 𝜖 = 0.7

3125-Mar-20

CONCLUSION• Reinforcement Learning using SARSA() is capable of• SARSA() is able to learn from its working environment andadapt accordingly to achieve near-perfect node level energyneutrality.

• Inclusion of weather forecast information helps in achieving

node level energy neutrality

EMSOFT 2017, SEOUL

attaining near-perfect node level energy neutrality.

32

THANK YOU FOR
LISTENING
shaswot@hal.ipc.i.u-tokyo.ac.jp

25-Mar-20

Battery isovercharged

100

SARSA Policy
Offline Policy

60
Constant
Duty Cycle

40
Battery (%)

Naive Policy
200278EMSOFT 2017, SEOUL

80

EnergyHarvested (%)

282

287

292

297

Time (Day)30230734

25-Mar-20

EnergyDuty Cycle (%) Battery (%) Harvested (%)

EMSOFT 2017, SEOUL

Percentage (%)

100806040

2001

481216Time (Hour)

202435