21-December-2021

Billions of Autonomous Sensor Nodes• Energy Harvesting Wireless Sensor
Nodes (EHWSNs)• sense, process and transmit data
wirelessly• harvest energy from the ambientenvironment

Autonomous
PerpetualOperation• Millions of usage
scenarios(deploy-and-forget)• Massive scaling

Air Temperature

Soil Temp

https://development.libelium.com/smart-agriculture-xtreme-sensor-guide/

Solar Irradiance

Leaf Wetness
Soil Moisture

Machine-to-Machine connections (billions)

Source: Cisco Annual Internet Report, 2018-2023

2

Energy Neutrality and multiple objectives
Energy Neutral Operation• Balancing energy generation andconsumption
Modern EHWSNs are multi-task• Sensing• Processing• Communication• Allocate energy among tasks• To maximize utility• According to user priority• Only known during runtime (not apriori)[Nakamura et al 2017, SenStick: Comprehensive
Sensing Platform with an Ultra Tiny All-In-One Sensor
Board for IoT Research]

Node Utility
Task 1

Task 2ENO

Task 3

Total Consumption

Savings

Total Available Energy
3

Optimizing over multiple objectives• Use Multi-objective Reinforcement Learning (MORL) tolearn the energy management policy• General Multi-Objective MDP Formulation

Proposal• Continuous States and Actions• Single/Multiple rewards• Novel low-compute MORL algorithms• Near-optimal policiesResults• Better than scalarization methods• Low learning costs• Runtime Tradeoffs

4Single-objective RL (SORL)
State (𝒔𝒕 )time

ENVIRONMENT

harvested energy
battery
predicted energy

SORL Agent(DDPG)

Action(Duty Cycle)𝑎𝑡 ∈ [0.1, 1]
RewardFntask energy req

Reward (𝑟𝑡 )Next State (𝑠𝑡+1 )Shresthamali et. al., Adaptive power management in solarenergy harvesting sensor node using reinforcement learning,
EMSOFT 2017.
Shresthamali et. al., Power Management of Wireless Sensor
Nodes with Coordinated Distributed Reinforcement
Learning,ICCD 2019.

5

Multi-objective RL (MORL)
State (𝒔𝒕 )time

ENVIRONMENT

harvested energy
battery
predicted energy
taskpriority
prioritytask
tasktaskpriority(𝜔𝑡𝐶req)
energytask energy reqtask energy req

MORL Agent(DDPG)

Action(Duty Cycle)
𝑎𝑡𝐴 ∈ 0.1, 1
𝑎𝑡𝐵 ∈ [0.1, 1]
𝑎𝑡𝐶 ∈ [0.1, 1]

RewardFunc. A

RewardFunc. B

RewardFunc. C

Reward (𝑟𝑡𝐴 )Reward (𝑟𝑡𝐵 )Reward (𝑟𝑡𝐵 )Next State (𝑠𝑡+1 )Node utility, wt = 𝜔𝑡𝐴 × 𝑟𝑡𝐴 + 𝜔𝑡𝐵 × 𝑟𝑡𝐵 + 𝜔𝑡𝐶 × 𝑟𝑡𝐶6

3. Optimizing overMultiple Objectives

https://www.fm-magazine.com/news/2019/nov/resource-allocation-best-practices-201922378.html

7

Challenges of learning with many rewards
Scalarization
Mix rewards into one scalar• Aoudia et al., 2018:• Multiply rewards together• Confounds rewards (noisy interference)• Ferreira et al., 2018:• Add rewards together• Tradeoffs not possible• Prior knowledge of relative weights
requiredTx
RewardSense
RewardProcess
RewardSingle Scalar Reward• Hsu et. al., 2014:• Complicated unintuitive reward function• Unexpected behavior (reward hacking)9Proposed MORL Framework
1. General Multi-Objective MDP Formulation• Rewards• Simpler, precise, intuitive rewards• Multiple sources of rewards (reward vector)• States• Continuous• Inclusion of temporal information (more Markov)

For efficientlearning• Actions• Continuous• Relative actions (Safe actions)

2. Low compute MORL algorithms (based on DDPG)• Runtime MORL

[Lillicrap et. al., 2015]• Runtime tradeoffs using pre-learned greedy policies• Off-policy MORL
• Learn tradeoff policies from scratch with low learning costs

10

Simulation Parameters
Component

RatingBattery (Rechargeable Li-Ion)

2000 mAh

Solar Panel

100 mA
Foldable Solar panel(100 mAh)

DeviceRatingWaspmote Sensor Platform
ATmega1281@15 MHz)

17 mA

Comm. Device (Zigbee 3)

40 mA

Sensor (GPS)

32 mA

TOTAL

89 ~ 100 mA

https://development.libelium.com/waspmote-technical-guide/
https://weffsas.com/shop/lithium-ion-battery-3-7v-2000mah/

Li-Ion Battery(2000mAh)

Waspmote11Experimental Results• Near-optimal Performance• Superior to scalarization methods• Dynamic Runtime Tradeoffs• Low learning cost
• Safer Exploration• Faster learning

12

Multiple Objectives
Energy Neutral Performance (ENP)

Sensing and Transmission

13

Energy Scheduling• At each timestep, the node allocates energy budget for• Sensing (w.r.t. user requirements)• Transmission (w.r.t. channel conditions)• By taking the user-defined relative priority (𝜔) between• Sensing (𝜔𝑠𝑒𝑛𝑠𝑒 )• Transmission (𝜔𝑡𝑥 )• Energy-neutral performance (𝜔𝐸𝑁𝑃 )• While ensuring long-term energy neutrality• Lower Downtimes = Better Energy Neutrality

14

Runtime Tradeoffs (2-tasks)
Off-policy MORL Algorithm• Tradeoff between sensing, transmission and energy-neutrality
Increasing 𝜔𝑠𝑒𝑛𝑠𝑒 increases sense-utility correspondingly:
red → green → orange → blue

15

Runtime Tradeoffs (2-tasks)
Off-policy MORL Algorithm• Tradeoff between sensing, transmission and energy-neutrality• 𝜔 = (𝜔𝑠𝑒𝑛𝑠𝑒 , 𝜔𝑡𝑥 , 𝜔𝐸𝑁𝑃 )Less priority
for ENO

More priority
for ENOHigher 𝜔𝐸𝑁𝑃 (green/red)ensure lower downtimes
than lower 𝜔𝐸𝑁𝑃(blue/orange)

16

Runtime Tradeoffs (2-tasks)
Off-policy MORL Algorithm• Tradeoff between sensing, transmission and energy-neutrality• 𝜔 = (𝜔𝑠𝑒𝑛𝑠𝑒 , 𝜔𝑡𝑥 , 𝜔𝐸𝑁𝑃 )Less priority
for ENO

More priority
for ENOHigher sensing utility
is achieved by trading off withlower energy neutrality(i.e., higher downtimes)

17

Runtime Tradeoffs (2-tasks)
Off-policy MORL Algorithm• Tradeoff between sensing, transmission and energy-neutrality• 𝜔 = (𝜔𝑠𝑒𝑛𝑠𝑒 , 𝜔𝑡𝑥 , 𝜔𝐸𝑁𝑃 )Less priority
for ENO

More priority
for ENO

Better energy neutrality(i.e., higher downtimes)
is achieved by trading off withlower sense-utility

18

Conclusion• EHWSNs require RL based methods for• Adaptive, scalable policies• RL is difficult for EHWSNs due to• Difficult problem formulation• High learning costs• High computation costs• Multiple objective optimization problem• Traditional MDPs and scalarization methods• Are sub-optimal• Have high-learning costs• Cannot tradeoff• Require complicated reward functions• Proposed MORL framework• Can learn near-optimal policies• With low learning costs• Can tradeoff at runtime• Simple/precise and diverse rewards can be used

19

Shaswot SHRESTHAMALI
https://www.shaswot.com

Masaaki KONDO

Hiroshi NAKAMURA

https://www.acsl.ics.keio.ac.jp/

http://www.hal.ipc.i.u-tokyo.ac.jp20

Thank You
Your questions/comments and feedback aremost welcome

21