Adaptive Power Management forEnergy Harvesting Sensor Nodes using
Reinforcement Learning
Shaswot Shresthamali, Masaaki Kondo, Hiroshi Nakamura
The University of Tokyo

CONTEXTEnergy Harvesting Sensor NodesSensor Node(capable of varying the duty cycle)

http://www.libelium.com/resources/ima
ges/content/products/plugsense/details/solar_powered_photo.png+
Battery+
Energy Harvesting Module
Theoretically capable of perpetual operation

8 March 2017

NAKAMURA LABORATORY

3

Challenge I
Say your battery is at 75% and there is plenty of sunshine
Do you◦ Use the solar power to charge your battery only◦ Use the solar power to charge your battery and drive thesensor node. If so, then with what proportion?
80018000

700

16000

Policy 1
Policy 2
Policy 3
Energy Harvested

BATTERY

600
14000

500

12000
10000

400

8000

300

6000
200
4000
100

2000
0

ENERGY HARVESTED

20000

01

2345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24TIME
8 March 2017

NAKAMURA LABORATORY

4

Challenge II

Environmental Sensor Networks – P.I. Corke et. al.

MOVING SENSORS
https://sites.google.com/site/sarmavrudhula/home/research/energy-management-of-wireless-sensornetworks

DIFFERENT ENVIRONMENTS

DIFFERENT SENSORS
http://www.mdpi.com/sensors/sensors-12-02175/article_deploy/html/images/sensors-1202175f5-1024.png

8 March 2017

NAKAMURA LABORATORY

5

Challenge III
BILLION AND TRILLIONS OF NODES

https://bendeetech.files.wordpress.com/2015/12/theinternetofthings2-540x334.jpg?w=350&h=200&crop=1

8 March 2017

NAKAMURA LABORATORY

6

Challenges II and III
When dealing with TRILLIONS of sensor nodes,
Customizing each node is impractical, impossible◦ Nodes should OPTIMIZE themselves.
◦ Nodes should ADAPT to their changing environments.

ENERGY HARVESTING NODES NEED TO BE

ADAPTABLE
SELF CALIBRATING
8 March 2017

NAKAMURA LABORATORY7

What this presentation is about
To demonstrate how to overcome the challenges byusing Reinforcement Learning (RL)
◦ Brief introduction to Reinforcement Learning◦ Our approach using RL◦ How this strategy performs compares to other methods◦ How this strategy adapts to changing environment

8 March 2017

NAKAMURA LABORATORY8OBJECTIVES

Objectives
Energy Neutral Operation (ENO)◦ Energy consumed = Energy harvested

Maximize Performance◦ Maximize Duty Cycle

Minimize Battery Downtime◦ Battery should never drop to zero

Minimize Energy Waste◦ Battery should not overcharge
Energy Waste = Energy Harvested–Energy consumed by Node–Energy to charge battery
8 March 2017

NAKAMURA LABORATORY

10SYSTEM MODEL

System Model
Solar Energy(Ideal)
Solar
Panel

Battery

SensorNode

Duty Cycle

Battery Reserve Level

Adaptive
Power
Manager

Energy being harvested
8 March 2017

NAKAMURA LABORATORY

12REINFORCEMENTLEARNING (RL)A BRIEF INTRODUCTION

What is RL
Type of Machine Learning - Learns by interactingwith Environment
Suited for Sequential Decision Making Tasks
Map situations (states) into actions – receive as
much reward as possible
Based on iterative process of trial and error – similar
to how humans learn. (Search and Memory)
8 March 2017

NAKAMURA LABORATORY

14

Why Reinforcement Learning
By using RL, it is possibleoTo optimize nodes with raw high level data and minimal
human input.
oTo adapt to changes in the environment parameters.

8 March 2017

NAKAMURA LABORATORY

15

Reinforcement Learning
What action
should I take toaccumulate total
maximum
reward?

OBSERVATIONS: Battery Level
Energy Harvested

REWARD, New State

Agent(Power Manager)

ACTION: Choose Duty Cycle

Environment
http://www.canstockphoto.com/gogreen-icons-concept-tree-12796260.html

http://wedreamabout.com/product/bb-8-droidthe-coolest-star-wars-toy-ever

8 March 2017

NAKAMURA LABORATORY

16

Reinforcement Learning
The question is:
WHICH ACTION TO TAKE WHEN YOU ARE IN A GIVEN STATE?

EXAMPLE:
Lots of sunlight | Battery at 60%

Do you[1][2]

drive the sensor node at full strength without
recharging?
drive the sensor node at half strength with partial
charging?
[1] https://spaceplace.nasa.gov/sun-corona/en/[2] https://handyenergy.ru/

8 March 2017

NAKAMURA LABORATORY17

Q- Value
Assign every state-action pair → Q-Value, Q(s,a)
Q(s,a) means if the agent• Starts from state s• Takes action a• Q(s,a) is the total reward it can expect in the best case scenario
Higher the Q-value,
better the action forthat particular stateStateX

Action 3

Action 1
Action 2

Q(X,1)
8 March 2017

Q(X,3)
Q(X,2)NAKAMURA LABORATORY

18

Q-Learning
Challenge → Determining the Q-Values for all stateaction pairs.

Q-table -> contains Q-Values of all possible stateaction pairs
Accomplished by Q-Learning Algorithm◦ Q-values are learned by interacting with environment.
◦ Iterative Process◦ Bootstrapping approach

8 March 2017

NAKAMURA LABORATORY

19

Q-Learning Algorithm
Q-Learning Algorithm◦ Use arbitrary estimates for Q-values◦ Use these estimates to decide on actions◦ Update Q-table by using the rewards received◦ Repeat until Q-value sufficiently converge8 March 2017

NAKAMURA LABORATORY

20EXPERIMENTS ON
ADAPTIVE POWER
MANAGEMENTUSING Q-LEARNING

State SpaceState is defined by :•amount of battery remaining• 200 possible levels•amount of energy harvested• 5 possible levels

Total possible states:

8 March 2017

200 x 5 = 1000

NAKAMURA LABORATORY

22

Action Space
Action: Choose duty cycle of the sensor node𝐴 = 𝑎 𝑡𝑘 ∈ 10%, 20%, 30% … .100%

8 March 201710%50 mW

50%

250 mW100%

500 mW
NAKAMURA LABORATORY

23

Reward Function
The reward depends on:• Distance from energy neutrality at time tk∆𝑒𝑛𝑒𝑢𝑡𝑟𝑎𝑙 𝑡𝑘 = 𝑒ℎ𝑎𝑟𝑣𝑒𝑠𝑡 𝑡𝑘 − 𝑒𝑛𝑜𝑑𝑒 𝑡𝑘• Amount of battery remaining

8 March 2017

NAKAMURA LABORATORY

24

RESULTS

Training and Testing
Training: Tokyo (2000 to 2009)

Testing : Tokyo (2010)◦Compare it with other methods.
◦Adaptation to diurnal and seasonal variations.
◦ Greedy and ε-greedy Implementations

8 March 2017

NAKAMURA LABORATORY

26

Comparison with other methods
100%

Higher Efficiency

90%

𝐴𝑐𝑡𝑢𝑎𝑙 𝐷𝑢𝑡𝑦 𝐶𝑦𝑐𝑙𝑒
𝐴𝑐ℎ𝑖𝑒𝑣𝑎𝑏𝑙𝑒 𝑀𝑎𝑥𝑖𝑚𝑢𝑚 𝐷𝑢𝑡𝑦 𝐶𝑦𝑐𝑙𝑒

80%
70%
60%
50%

Efficiency(%)

40%

Energy Wasted(%)

30%
20%
10%

Lower Waste0%Naïve

Duty Cycle is
proportional tobattery level
8 March 2017

Kansal

Our method
using RL

𝑇𝑜𝑡𝑎𝑙 𝐸𝑛𝑒𝑟𝑔𝑦 𝑊𝑎𝑠𝑡𝑒𝑑
𝑇𝑜𝑡𝑎𝑙 𝐸𝑛𝑒𝑟𝑔𝑦 𝐻𝑎𝑟𝑣𝑒𝑠𝑡𝑒𝑑
𝐸𝑛𝑒𝑟𝑔𝑦 𝑊𝑎𝑠𝑡𝑒= 𝐸𝑛𝑒𝑟𝑔𝑦 𝐻𝑎𝑟𝑣𝑒𝑠𝑡𝑒𝑑−𝑁𝑜𝑑𝑒 𝐸𝑛𝑒𝑟𝑔𝑦− 𝐶ℎ𝑎𝑟𝑔𝑖𝑛𝑔 𝐸𝑛𝑒𝑟𝑔𝑦

Fix duty cycle forpresent day by
predicting totalenergy for next day
NAKAMURA LABORATORY

27

ADAPTATION TO
SEASONAL
CHANGES
8 March 2017

NAKAMURA LABORATORY

28

Performance in Summer

High Duty Cycle evenduring the night

8 March 2017

NAKAMURA LABORATORY

29

Performance in WinterLower duty cycle during the night

Duty Cycle (%)
100%Harvested
Energy (%)

80%
60%
40%20%0%1360

1400
Battery (%)

8 March 2017

1440

1480

EPOCH

NAKAMURA LABORATORY

30

ADAPTATION TO
CHANGE IN
LOCATION
8 March 2017

NAKAMURA LABORATORY

31

Implementation: ε-greedy approach
Perfect Q-convergence takes too long.
Instead, use ε-greedy approach with non-converged
Q-table.
ε-greedy approach:◦ Take the best action by default.
◦ Take a random action with probability ε.
◦ Increasing ε → Exploration◦ Decreasing ε → Exploitation
8 March 2017

NAKAMURA LABORATORY

32

Adaptation to change in climate•Wakkanai (very littlesunshine)•Compare between• a greedy approach (Offline)and• an -greedy approach(Online).

•Training: 2000-2009 Tokyo•Testing: 2010 Wakkanai

8 March 2017

NAKAMURA LABORATORY

33

Adaptation to change in location
Total number of times thebattery was completely
exhausted

Average Duty Cycle (%)

65
64
63

Greedy (Non adaptive)

62
61

(8)

60
59

(14)

(14)

58

-greedy (adaptive)

57
56
55

With -greedy implementation, the agent adapts to theenvironment and minimizes instances of battery exhaustion.
201020112012

Wakkanai Offline
8 March 201720132014

2015

Wakkanai Online

NAKAMURA LABORATORY

34

With and Without Forecast Information

8 March 2017

NAKAMURA LABORATORY

35

CONCLUSION

CONCLUSION•Proposed system is able to meet objectives of• Energy neutrality• Maximizing performance•Exceeds the performance of other schemes•Capable of adaptation•Inclusion of weather forecast results in smarter
operation
8 March 2017

NAKAMURA LABORATORY

37

THANK YOU FOR
LISTENING
ANY COMMENTS OR QUESTIONS ARE WELCOME