Power Management of Wireless Sensor Nodes
withCoordinated Distributed Reinforcement Learning
SHASWOT SHRESTHAMALIMASAAKI KONDOHIROSHI NAKAMURA
THE UNIVERSITY OF TOKYO
2 December 2019ICCD 2019, Abu Dhabi

Wireless Sensor Networks for Internet of Things

Energy Harvesting Wireless Sensor Nodes (EHWSNs) are
an attractive solution for Internet of Things (IoT).
• Autonomous operation• Perpetual operation
Requires scalable , intelligent and adaptive power
management policies
http://www.libelium.com/wp-content/themes/libelium/images/content/applications/libelium_smart_world_infographic_big.png

2

ENO-RL System3Energy Neutral Operation (ENO):• harvested energy equals energy spent• i.e., perpetual operation

ENO-RL OBJECTIVE• Minimize battery violations• overflows (100% battery)• downtimes (0% battery)• Maximize utility (duty cycle)• Sensor is always ON𝐵𝑀𝐴𝑋

Batt. Profile #1
Batt. Profile #2
Batt. Profile #3

Overflows

𝑏𝑜𝑝𝑡 ± 𝑏𝑚𝑎𝑟𝑔𝑖𝑛➢ Solar EHWSN➢ Duty cycle determines nodeenergy consumption➢ Hourly data
➢ Tokyo: 1995-2018} ENO𝐵𝑀𝐼𝑁
Solar EnergyDowntimes

Reinforcement Learning (RL)
• Assuming a Markov Decision Process,1. Perception: What is the state of the agent?
2. Action: Take an action according to a policy that maps
states to actions
3. Reaction: Receive a reward as feedback from theenvironment
4. Learning: Learn from the reward to refine the policy.
• OBJECTIVE: Accumulate as much reward as possible
What is the“goodness” or “value“of taking this action atthis state?

4

Deep Reinforcement Learning

5• Use neural networks to predict the state-action value• Learning via boot-strapping (better estimates from estimates)

Approximated withparameters 𝜽State-Action
Pairs: (𝑠, 𝑎)Function,𝑓(𝑠, 𝑎, 𝜃)
CriticExpected Returnfor action 𝐴

Expected Returnfor action 𝐵

Dueling Double Deep Q-Networks
Wang, Ziyu, et al. "Dueling
network architectures fordeep reinforcement
learning." arXiv preprint
arXiv:1511.06581 (2015).

State, sChoose
ActionREWARDActor

𝜖 -greedy• Always take greedy action• With probability 𝜖, take random
action• Start with high 𝜖 and decrease
gradually (𝜖 - annealing)

Single Agent ENO-RL: B-ENO

6

Basic-ENO: Naïve implementation of Deep RL for ENO-RLwith a single agent.
State Space:

Action Space:-• Discrete Duty Cycle,- 𝐷𝑚𝑖𝑛 ≤ 𝑑𝑡 ≤ 𝐷𝑚𝑎𝑥

Battery level
Harvested Energy
Energy Neutral Performance
Weather Forecast

Low Rewards
High Rewards
Low Rewards

https://favpng.com/png_view/wireless-cliparts-wireless-sensor-network-internet-of-things-clip-art-png/JKvwgxaM

B-ENO: Performance
PERFORMANCE METRIC (Energy Neutral Operation)
ENO is achieved if there are less than 24 violations in 365
consecutive days
LEARNING TIME
(Time required to achieve ENO (1995-)

90,186 hours (~10 years)

LEARNING PENALTY(Violations committed during learning)

17,722 violations (~2 years)
OPERATION PENALTY(Violations committed during greedy implementation from 1995-2018)0 violations

7

Accelerating B-ENO
Can we decrease the learning time and penalty byleveraging• the multiple nodes of the sensor network to• simultaneously interact with the environment

Distributed RL (DiRL)• Leverage the collective experience to learn better and faster• Explore wider and faster

experience => (present_state, action, reward, next_state)8

Distributed RL (DiRL)
10 agents (nodes) and 1 central learner

ServerEvery day(episode):
1. Nodes execute e-greedy policy
2. Upload experiences to central server
3. Server executes 𝑁𝑙 learning steps
4. Server broadcasts new policy to nodes
https://favpng.com/png_view/cloud-hosting-cliparts-web-server-cloud-computing-web-hosting-service-icon-png/BUiv4YG7

9

Challenges and Proposed Methods
1.

10

Use Distributed RL (DiRL) to accelerate learning(Distributed-ENO)
 State space is not fully explored -> non-robust policies

2.

Partition the state-space via coordinated exploration(Partitioned-ENO)
 Some agents face higher risks of violating ENO

3.

Uniformly distribute the risk of exploration amongnodes (Safe-ENO)
 Playing safe does not give best performance.

4.

Dynamically adapt the exploration rate to tradeoff
between learning time and learning cost (Adaptive-ENO)

Comparative Analysis
Algorithm

11Learning
Time

Learning
Penalty

Operation
Penalty(time to reach ENO)(# of violations)(# of violations)

10 yrs17,7220

Learning time andpenalty too high

0.6 yrs

7,930

20

Insufficient state
space exploration

0.4 yrs

8,817

16

State-spacepartitioning
distributes risk
non-uniformly

0.3 yrs

5,392

80.2 yrs

5,9210B-ENO(basic)D-ENO(naïve distributed)(safe)

A-ENO(adaptive)*Penalties are summed across all nodes for D,P,S,A-ENO

BetterS-ENO

Safer(partitioned)

FasterP-ENO

Comments

Tradeoff between
learning time andpenalty
Trading off learning
costs dynamicallyComparison: Learning Time

12

Comparison: Learning Penalty

13Partitioning the State Space• High correlation
between duty cycles(actions) and battery
levels (states)

Explored using
high duty cycle

14
Explored using
low duty cycle• Bias node duty cycles
to explore different
battery states• Coordinate the nodesof DiRL to explore
different regions of thevast problem statespace

Partition the state space using
different exploratory actions.

Comparison: Exploration

15Less spread as a result ofnaïve 𝜖 – greedy exploration.

More spreadin state spaceresulting in
better learning

A-ENO: Year Run✓ Battery level is well within the required range.
✓ No violations despite seasonal and diurnal variations

16

A-ENO: Seasonal Adaptation

17Summer✓ During summer, highduty cycles are usedto maximize utility.
✓ Adaptation to sudden
low energy daysWinter✓ During winter, lower
duty cycles are used
to save energy.
✓ Duty cycle ismaximized whenpossible.

CONCLUSION

181.

Non DiRL solutions are optimal but take
impractically long to learn (B-ENO).

2.

DiRL solutions learn faster but naïve
implementations are sub-optimal (D-ENO).

3.

Learning cost and time can be decreased bypartitioning state space exploration (P-ENO).

4.

Partitioning state space distributes risk nonuniformly which can be traded off for some
performance loss (S-ENO).

5.

Dynamically adjusting exploration rate trades off
risk and performance and enhances learning (AENO).

19

Thank you
Any Questions or Comments

B-ENO: Learning

20

D-ENO: Learning

LEARNING TIME
5,249 hours (~0.6 years)
LEARNING COST (cumulative)
7,930 violations (~0.9 years)

21

D-ENO: Testing22

Challenges in Deep RL for ENO-RL• Requires LOTS of training data

 Longer training periods
 Larger number of violations(downtimes and overflows)• Unstable learning due to23

generates
POLICYEXPERIENCE
refinesbootstrapping Training should include a “correct”
mix of positive and negative
GOALS:experiences.
 Maximizing EXPLORATION of the - Converge to a robust policy- Minimize learning timestate-space is critical Unseen states may cause the network to- Maximize node utility
destabilize(minimize violations during• Also, maximize utilitylearning) Exploration-exploitation tradeoffSafe Exploration: S-ENO24• High-duty cycle as non-greedy action -> more violations• Low-duty cycle as non-greedy action -> less violations• Change the preferred non-greedy action after every episode.
Day 1

Day 2

Day 3

Day 4

Day 5…

Node 1𝑑1𝑑2𝑑3𝑑4𝑑5…

Node 2𝑑2𝑑3𝑑4𝑑5𝑑6…

Node 3𝑑3𝑑4𝑑5

𝑑6𝑑7

…

Node 4𝑑4𝑑5𝑑6𝑑7𝑑8

…

…

…

…

…

…

…

…

Adaptive exploration: A-ENO

25• Different nodes -> different environments• Different environments -> different learning behavior• Different learning behavior -> different annealing rates for 𝜖.

• Increase 𝜖 if reward is negative.
• Decrease 𝜖 if reward is positive.

Adaptive exploration: A-ENO

More diverse
experiences in thebeginning

Robust performancefor anomalous states

26