REINFORCEMENT LEARNING FOR POWER MANAGEMENT IN
SOLAR ENERGY HARVESTING SENSOR NODES*
Shaswot SHRESTHAMALI, Masaaki KONDO, Hiroshi NAKAMURA
Graduate School of Information Science and Technology, The University of TokyoINTRODUCTION
Internet of Things (IoT) and Pervasive Computing requires billions and trillions of sensor nodes. Energy Harvesting Wireless Sensor Nodes (EHWSN) play a critical role in forminga sustainable, maintenance-free network of perpetually communicating autonomous devices for the IoT infrastructure. Energy autonomy (neutrality) of the sensor nodes
should be ensured for perpetual operation. Here we consider the case of maximizing the utility of a solar energy harvesting sensor node while maintaining energy neutrality.

CHALLENGES
1. Energy Neutral Operation

2. Adaptivity

3. Diversity and Scaling• Amount of energy harvested should equal

The node has to be able to adapt to changes in• The solution should be feasible for sensor nodes

the amount of energy consumed by the node.

regardless of their functionality or working

energy harvesting profile due to• A minimum duty cycle should be maintained at
all times.
• The battery should never be completely full orenvironment.

-Diurnal /Seasonal Variations-Climatic Variations

environment should not compromise the node’s-Changes in device parameters

performance• Anychanges

fromtheoriginal

workingdepleted.

REINFORCEMENT LEARNING AS A SOLUTION
 Hand-engineered solutions will not be practical

PERCEPTIONACTIONowing to the diverse scenarios of application andGet STATE from
- Battery Level- Solar Energy- Weather Forecast

Choose a DUTY CYCLE
for the sensor node byreferring to the Qtable

UPDATE the Q-tableusing SARSA(𝜆).

Receive REWARD
according to theReward Function

LEARNING

FEEDBACKthe sheer population of sensor nodes.
 We propose a one-size-fits-all solution i.e. a power
manager for EHWSN that is capable of✓ Learning the best power management strategy
through a context aware action-perception
learning cycle.
✓ Adapting to any changes in its environment
 We

useaReinforcement

Learning(RL),
Figure 1: System ModelSARSA(𝝀) with a system model shown in Figure 1.

EXPERIMENTAL RESULTSTESTING:

Tokyo, 2010
 Tokyo, 2011
 Wakkanai, 2011

Tokyo, 2011

604020DAY 290

 Wakkanai has a much colder climate

80

Near Perfect Node Level
Energy Neutrality

48121620

Optimal Battery
Battery (%)
Level, B0(Offline Policy)

Percentage (%)

TRAINING:

Percentage (%)

representative days from 2010 and trained the agent for
10,000 iterations on each of them.

EnergyBattery (%)(SARSA Policy) Harvested (%)

Optimal Battery
Battery (%)
Level, B0(Offline Policy)

We ran simulations with historical weather data. We took 1224

Wakkanai, 2011
60

40
20035

3637

38

3940

41

42

43

44

45

46

47

48

49

Time (Day)Time (Hour)compared to Tokyo

EnergyBattery (%)(SARSA Policy) Harvested (%)

Figure 3: Adaptation to change in location

Figure 2: SARSA(𝝀) achieves near optimal
performance

We compare the performance of SARSA(𝝀) with an Offline

Figure 2 shows the battery profiles for SARSA(𝜆)Figure 3 shows the battery profiles for Wakkanai, 2011.

Policy. Offline Policy is calculated using a linear optimizer

and Offline Policy. SARSA(𝜆) is able to achieve near

The agent is able to adapt to Wakkanai climate evenassuming a perfect omniscient weather predictor.

perfect node level energy neutrality.

thought it was trained with Tokyo data.

Optimal Battery
Battery (%)
Level, B0(Offline Policy)

EnergyBattery (%)(SARSA Policy) Harvested (%)

65

AUTUMN 2011, Tokyo
60
40200332333334335336337338339

Time (Day)
Figure 4: Adaptation to seasonal variation

Percentage (%)

6430100100

80

60

4020
Tokyo
31 October 2011

32463322

62

320

61

318

60

31659

314

58

50

312070310

20Large deviation from optimal battery level
when not including weather forecast

308

Percentage (%)

40

Percentage (%)

60

Energy Harvested (%)*watermarked, dashed lines correspond to the case of normal (full) solar capacityEnergy
Battery level withBattery level without
Weather Forecast (%) Weather Forecast (%) Harvested (%)

SPRING 2011, Tokyo

Battery (%)

Duty Cycle (%)

0Time (Day)

14

8

12

16

2024Time (Hour)

Figure 5: Inclusion of Weather Forecast

Figure 6: Adaptation to change in device parameters

Figure 4 shows how the agent is able to account for seasonal

Figure 5 shows that inclusion of weather forecast inIn Figure 6, the agent is able to achieve energy

differences in harvested energy and maintain energy neutrality.

decision-making results in better performance.

neutrality even though the solar capacity is halved.

*This work was partially supported by JSPS KAKENHI Grant Number 16K12405.