REINFORCEMENT LEARNING FOR POWER MANAGEMENT IN
ENERGY HARVESTING SENSOR NODES*Shaswot SHRESTHAMALI, Masaaki KONDO, Hiroshi NAKAMURAGraduate School of Information Science and Technology, The University of TokyoINTRODUCTION
In the near future, Internet of Things (IoT) will consist of billions and trillions of nodes. Energy Harvesting Wireless Sensor Nodes (EHWSN) play a critical role in forming asustainable, maintenance-free network of perpetually communicating autonomous devices for the IoT infrastructure. Energy autonomy (neutrality) of the sensor nodes
needs to be ensured for perpetual operation. Here we consider a case of a solar energy harvesting sensor node.

CHALLENGES
1. Energy Neutral Operation

2. Adaptivity

3. Diversity and Scaling

The node needs to be energy neutral. The amountThe node has to be able to adapt to changes inWhen we deal trillions of different sensor nodes,of energy harvested should equal the amount ofenergy harvesting profile due tocustomizing the power management policy for eachenergy consumed by the node. This way we cannode according to its power rating and environment-Diurnal Variationsensure that battery will never go empty and that-

Seasonal Variations

none of the harvested energy goes to waste. This is-Climatic Variations

changes. Instead, we need a one-size-fits-all

not a trivial issue because the energy harvested is-Changes in device power consumption

solution that can work for all types of EHWSN-Battery Degradation

scenarios.

often unreliable and unpredictable.

will be incredibly impractical. Moreover, the policy
may have to revised if their working environment

REINFORCEMENT LEARNING AS A SOLUTION
Since heuristic policies will not suffice, the most

SIMULATION PARAMETERS

practical solution is to use a context aware Battery: 40,000 mWh (ideal)

perception-action cyclic approach. Our solution

 Node power consumption: 100 mWh to 500

is to use Reinforcement Learning (RL). The powermWh (5 discrete duty cycles)

manager learns the power management policy byinteracting with the environment and memorizing Solar power: 0 to 3000 mWh (Energy profiles

the best strategy after numerous hit-and-trial

obtained from Japan Meteorological Agency)

interactions.

 Weather Forecast: 6 different types (Very little- States: Defined by battery level, harvested

sun, Overcast, Partly Cloudy, Fair, Sunny, Veryenergy, weather forecast and energy neutral
Figure 1: System Modelperformance (ENP)Sunny) Training: Solar data for Tokyo, 2010

- Actions: Choose a duty cycle

We use a simplified system model that consist of a

- Reward: Received at end of day (episode)solar panel, an ideal battery, a sensor node and adepending on net energy difference Testing: Solar data for :RL power management unit that specifies the duty

- Learning: SARSA -  with eligibility traces

Wakkanai, 2011 Tokyo, 2011

cycle of the sensor node.

EXPERIMENTAL RESULTS

Tokyo, 2011

60

20

Optimal Policy using
non-causal data

ProposedMethod

60
40
200

Our method comesvery close to the
optimal solution

Solar Energy Profile

DAY 29
4

812

16

20

20
0

58

59

60

61

62

6364

AUTUMN60
40

65

Distance fromenergy neutrality (%)(SARSA Policy)
Distance fromenergy neutrality (%)(Optimal Policy)

24

Time (Hour)

Figure 2: SARSA Method vs. Optimal Policy

In the above figure, we observe that bothpolicies have similar battery profiles. While thetwo policies may differ in how they allocate theenergy, at the end of the day, both of themcome very close to perfect energy neutrality.
The actual net deviation from the initial battery
level (60%) was only 491.875mWh and
191.875 mWh for SARSA and Optimal Policy
respectively.

00332

35333

334335336

337338

Our method is able to adapt to both spring and autumn
solar energy profiles. The resultant battery profile
differs very little from the optimum policy.
This is largely due to our novel RL problem formulation(state definition and reward function). Because of ourunique RL problem formulation, the power manager
takes energy neutral performance into account when
making decisions rather than only the battery reservelevel.

3839

40

41

42

43

44

45

46

47

48

49

Our proposed method adapts well in Wakkanai, a locationthat has a drastically different climate from the one it wastrained in.
Wakkanai 201170

Figure 3: Adaptation to Seasonal Changes

37

Figure 4: Adaptation to a new location

339

Time (Day)36

Time (Day)

EnergyHarvested (%)

20

ProposedMethod

20

Battery (%)(Optimal Policy)

40

4080

Battery (%)

6040

Battery (%)

80

Battery (%)(Proposed Method)SPRING

80

Percentage (%)

Duty Cycle (%)

100

60

Lower deviation when including
weather forecast information

With Weather
Forecast

50

Without Weather
Forecast

30
100Solar Energy
Profile
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324

Tokyo 2011

Optimal Policy

Battery (%)

We compare the performance of our proposed SARSA -  algorithm with an optimal policy. The OptimalPolicy uses non-causal information and linear optimization techniques to arrive at an optimum solution. We
compare the performance of the two policies in different seasons and locations. We also show how
including weather forecast information contributes to enhancing the performance of our method.

Wakkanai, 2011

Time (Day)
Figure 5: Inclusion of weather forecast enhances performance

Furthermore, we show that inclusion of weather forecast
information results in better results.

*This work was partially supported by JSPS KAKENHI Grant Number 16K12405.