Adaptive Power Management in Solar Energy Harvesting Sensor
Node using Reinforcement Learning
SHASWOT SHRESTHAMALI, The University of Tokyo
MASAAKI KONDO, The University of Tokyo
HIROSHI NAKAMURA, The University of Tokyo
In this paper, we present an adaptive power manager for solar energy harvesting sensor nodes. We use a
simplified model consisting of a solar panel, an ideal battery and a general sensor node with variable duty
cycle. Our power manager uses Reinforcement Learning (RL), specifically SARSA(λ) learning, to train itself
from historical data. Once trained, we show that our power manager is capable of adapting to changes inweather, climate, device parameters and battery degradation while ensuring near-optimal performance without
depleting or overcharging its battery. Our approach uses a simple but novel general reward function andleverages the use of weather forecast data to enhance performance. We show that our method achieves near
perfect energy neutral operation (ENO) with less than 6% root mean square deviation from ENO as comparedto more than 23% deviation that occur when using other approaches.
CCS Concepts: • Computing methodologies → Machine learning approaches; • Hardware → Energy
generation and storage; Power estimation and optimization; • Computer systems organization →Embedded and cyber-physical systems;
Additional Key Words and Phrases: Wireless sensor nodes, reinforcement learning, power management, IoT
ACM Reference format:
Shaswot Shresthamali, Masaaki Kondo, and Hiroshi Nakamura. 2017. Adaptive Power Management in Solar
Energy Harvesting Sensor Node using Reinforcement Learning. ACM Trans. Embedd. Comput. Syst. 17, 4,
Article 39 (October 2017), 22 pages.
DOI: 0000001.0000001

1

INTRODUCTION

Energy harvesting sensor nodes (EHSN) are sensor nodes equipped with an energy buffer (battery)
and an energy harvesting module. The presence of a battery along with an energy harvesting
module theoretically allows for perpetual operation limited only by the lifetime of the hardware[8]. Perpetual operation is critical for realizing pervasive computing and Internet of Things (IoT) as
it opens the possibility of autonomous operation of sensor nodes.
Before the integration of energy harvesting modules,optimization techniques revolved aroundminimizing power consumption of the node to extend the lifetime of the battery [20] and [22].
This work was partially supported by JSPS KAKENHI Grant Number 16K12405. The first author acknowledges the Japanese
Government (MEXT) Scholarship received for his study in The University of Tokyo.
Authors’ address: Nakamura Laboratory, Room 508, 5F, Engineering Building 1, The University of Tokyo, Hongo 7-3-1,
Bunkyo, Tokyo, JAPAN, 113-8656
This article was presented in the International Conference on Embedded Software 2017 and appears as part of the ESWEEKTECS special issue.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and thefull citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2017 Copyright held by the owner/author(s). Publication rights licensed to ACM. 1539-9087/2017/10-ART39 $15.00
DOI: 0000001.0000001
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39

39:2

S. Shresthamali et al.

However, with the possibility of harvesting energy from the ambient environment, the node is able
to recharge its battery. During times when energy cannot be harvested, the node can operate on
the energy stored in the battery.
This leads to the concept of energy neutral operation (ENO), introduced in [10], [16] and [9]. ENO
is achieved when the energy consumed by the node is less than or equal to the energy harvestedby the environment. In addition, we would also like to fully utilize all of the energy harvested to
power the sensor node. The condition when the amount of energy harvested equals the amount ofenergy consumed by the node is termed as node level energy neutrality[19] or ENO-Max condition[23]. Achieving node level energy neutrality ensures a minimum level of operation by the sensor
node at all times. In addition to this, the node is also able to increase its utility by exploiting as
much of the harvested energy when possible [9].
Achieving node level energy neutrality comes with several constraints, namely the size of thebattery and the maximum/minimum rates at which energy can be consumed and harvested. In
addition, the energy harvested from the environment is often unpredictable and unreliable. In sucha context, achieving node level energy neutrality is not trivial.
A major challenge in achieving node level energy neutrality is using the correct power management strategy to accommodate for the changing energy harvesting profiles. Sensor nodes suchas those strapped to animals or mobile sensor nodes will encounter diverse and varying energy
harvesting opportunities. In addition, changes in weather patterns and climate will also require thesensor nodes to adopt a suitable power management strategy. The sensor node must also be able
to adjust its behavior to account for changes in its device parameters such as changes in energy
harvesting efficiency, battery degradation [14] and partial node failure/decrease in node’s energy
efficiency. It is not practical to prepare heuristic contingency plans for all situations. This problem
is even more aggravated when we have to deal with billions and trillions of sensor nodes, each witha unique power consumption profile and deployed to a unique environment. A natural solution inthis case is to have sensor nodes that are capable of autonomously learning optimal strategies andadapting once deployed in the environment.
Several adaptive approaches to power management have been previously proposed. Section 2 goes
into more details about these methods. Our paper is based on these approaches and overcomes someof their limitations. The first formal solution was presented in [9]. The authors use optimization
techniques based on linear programming integrated with an energy prediction mechanism. In[23], the authors propose a linear quadratic control system for adaptive power management.
Reinforcement learning (RL) for power management is used in [5],[6], [2] and [15]. More details
on RL are given in Section 3. In short, in a RL setting, the power manager(or the agent) executes
some actions and interacts with its environment. The environment responds by awarding the agent
with a single value scalar reward according to some reward function. Through a number of suchinteractions, the agent searches its action space and memorizes (learns) the most favorable action
for a particular state the agent might be in. An RL based learning approach is adaptive by nature of
its learning [13]. Since its learning is integrated with interaction with its environment, the agent isable to respond to changes in slowly varying non-stationary environments [21].
We propose a power management policy based on tabular SARSA(λ) reinforcement learning (RL)[21] for a solar EHSN. In this method of RL, some fraction of the reward is back-propagated to all
the actions and states that contributed to its achievement. This back propagation of feedback resultsin faster learning [21] as compared to other methods that propagate their rewards by only one step.
In our formulation of the problem, the node is able to dynamically adapt its power consumption
depending on the energy harvesting opportunities by varying its duty cycle to ensure ENO. The
power manager makes its decisions based on information about its distance from energy neutrality,
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

Adaptive Power Management in Solar Energy Harvesting Node using ...

39:3

battery level, amount of energy being harvested and the weather forecast for the day. The learningtheory and its implementation methodology are explained in more detail in Section 4 and Section 5.
The distance from energy neutrality is referred here as Energy Neutral Performance (ENP). One ofour main contributions is the inclusion of ENP in the state definitions. This dramatically reduce thelearning time and enables the agent to be highly adaptive to environmental changes. Our results,
described in Section 6 show that the agent is able to adapt to seasonal variations, changes in climate
due to change in location, battery degradation and changes in device working parameters . As to
our current knowledge, the general adaptivity of a RL based power management strategy for EHSN
has not been investigated before.
The reward function is extremely critical in a RL framework as it is responsible for educating theagent on what kind of behaviors are favorable. [2] and [15] use metrics related to data transmission
as rewards. This makes sense when the power consumption of data transmission is significantly
greater than that of sensing operations. This assumption limits the scope of application and excludes
situations where other critical tasks consume significant power. For example, when a mobile sensor
node has to allocate power consumption between locomotion, processing, transmission and sensing,
the amount of data transmitted may not be a good basis to determine the optimality of the actions.
The reward function mentioned in [6] has a more general scope of application. Here, the author
propose a reward function based on the instantaneous battery level and ENP. In our paper, we
improve on this and introduce a simplified reward function that is natural, intuitive and performs
satisfactorily. We use the ENP at the end of an episode(day) as the sole basis of our reward function.
This is a novel contribution of our work. An added benefit to this is that this reduces the variance inbattery levels and this in turn contributes to increase in battery life [1]. The novelty of this reward
definition is that it is indifferent to the node’s tasks that consume power. As a result, our proposed
power management policy has a wider scope of application. In this paper, we assume all the power
consumed by the node is for sensing purposes. This is only for the sake of example and we can
extend this to any kind of node operation.
Our final contribution is to take into account the information about the weather forecast toenhance performance. [18] use forecast data to model a weather predictor. It is obvious that theavailability of a perfect weather oracle would greatly enhance the performance of a power manager.
[9] and [2] mention some power management strategies when such non-causal data is available.
We use the formulation in [9] to obtain the theoretical upper limit in performance and use it forcomparison. It is impossible to expect such non-causal information in practice. However, it ispossible to obtain some general indication of future weather for the day. The information about thetype of weather - for e.g. sunny, fair, overcast etc. can be easily acquired from weather websites andapps. We make use of this information to increase the performance of our agent. For the systempresented in this paper, we use the forecast data to give an estimate of expected solar energy.
However, the same argument can be made for other EHSN such as those driven by wind power.
In summary this work presents the following contributions:• Adaptivity: We propose a SARSA(λ) RL based power management approach to ensure ENOMax operation. This approach enables the power manager to adapt to changes in weather,climate, battery degradation and device parameters while achieving faster convergence.
This is attributed to our novel idea of using ENP as a state definition parameter.
• General Scope of Application: A direct consequence of our unique reward function isthat the power manager policy can learn to accommodate any kind of realistic system. As
a result, the scope of application is more general. Using our method, the policy can adapt
accordingly and maintain near-optimal performance even if the system parameters are tochange over time.
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:4

S. Shresthamali et al.
• Enhanced performance: Our power management strategy improves performance by asmuch as four times by leveraging information about the general weather forecast. The useof such data allows the agent to anticipate the amount of energy that can be harvested andadopt a suitable power management strategy.

2 RELATED WORK
An overview of different energy harvesting architectures, sources of ambient energy and realworld example of EHSN is presented in [19]. In [4], the authors elaborate on design challenges
and solutions for energy harvesting communication systems in predictable and unpredictableenvironments. [11] surveys management strategies for wireless sensor nodes on basis of theirenergy provision and consumption.
The first formal description of energy harvesting sensor nodes and ENO was done in [9]. The
authors in [9] present a linear programming optimization solution when non-causal information
about the environment is available. We adapt their optimization technique to determine the optimal
policy and compare it with our results. When non-causal information is not available, they predict
the amount of energy that is expected and decide on the duty cycle accordingly. They take into
account battery inefficiencies and also allow for adaptation of the duty cycle to accommodate forany changes in predicted and actual harvesting conditions. However this technique relies heavily on
the accuracy of its prediction mechanism and prior knowledge of the statistics of the environment.
This is not always a realistic assumption.
In [23], the authors approach this problem by specifying a battery-centric objective function.
They argue that by minimizing the average square deviation in battery level, ENO-Max conditionis satisfied. They achieve this by using a linear quadratic-tracker . They also take into accountthe minimization of duty cycle variance. They do not attempt to model the energy source and do
not require prior information about the environment. While this approach is adaptive in nature,
it requires careful calibration of hyper parameters depending upon the environment. The agent
would not be able to "fix" its own hyper parameters if its environment should change.
An adaptive duty cycling method using continuous time Markov Chain Modeling by taking
into account the Quality of Service (QoS) requirements and the rate of change of battery level
depending upon the battery chemistry is given in [3]. RL strategies for power management aredescribed in [17]. In [2], the authors tackle the case of point to point wireless communication systemwith stochastic data arrival and channel state processes. They provide optimization techniques
when non-causal information about the environment is known and in cases when statistics of theenvironment are known beforehand. They then propose an RL based learning theoretic approachto maximize total transmitted data. The authors use discrete states, actions and rewards similar to
our approach. In [15], the authors improve on this by using function approximation with RL to
accommodate for continuous states and rewards. As mentioned before, the techniques proposed in[2] and[15] use data transmission metrics as a basis for rewards. This is not always reasonable ingeneral applications. Fuzzy rewarding schemes are mentioned in [12] and [7].
Optimizing the duty cycle to achieve node neutrality using RL is proposed also in [5], [6]. Theirbasis for reward is the distance from energy neutrality and the current battery level. While this isa more general formulation of the reward function, it is quite handcrafted. The reward function
is an indication of what kind of states or actions are favorable rather than directives on how toachieve a certain goal. The distance from energy neutrality is not a good measure of reward during
an episode. We also use the same metric to determine the reward. However, the reward is fed backonly at the end of the episode. We believe that this reflects the ENO-Max objective better.
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

Adaptive Power Management in Solar Energy Harvesting Node using ...

39:5

None of the previous research have investigated the adaptivity of their algorithms to changes
in climatic conditions and battery degradation. We observe the behavior of our agent to various
environments and observe its adaptive performance.
[18] show that performance of EHSN can be improved by using weather forecast information.
Their work uses this information to make better predictions about the energy that can be harvested.
Along the same lines, we also leverage the use of easily obtainable forecast data for a particular
day to improve performance.
3

THEORETICAL BACKGROUND

In this section, we explain our system model. We then go briefly into RL and the SARSA algorithm.
3.1

System Model

Our system model contains four components - an energy harvesting source (solar panel), an energy
buffer (battery), a sensor node (load) and a power management unit that controls the node power
consumption as illustrated in Figure1. The battery has a maximum capacity of B MAX . We assume
the sensor node to have a variable duty cycle. The power management unit uses information about
the current battery level, energy being harvested and the information about the weather to decide
the duty cycle of the node. We further assume that a higher duty cycle implies a higher power
consumption and higher performance from the sensor node.
Renewable
Energy Source eharvest(tk)(Solar Panel)
Energy being
harvested

Energy Buffer(Battery)enode(tk)Battery Reserve Level

Sensor Node
(Load)

Duty Cycle

Adaptive PowerManager using RL
Weather
Prediction

Fig. 1. System Model

We use a discrete time model where time is divided into equal intervals called epochs. A setof epochs constitutes an episode. In the beginning of each epoch, the power manager receives
information about the weather condition that can be expected for the day. ebat t (tk ) is the batteryenergy level at the start of the epoch tk . During each epoch, tk , the system receives a finite amountof energy, eharvest (tk ), from the environment. The power manager decides on a duty cycle, d(tk ).
The sensor node consumes enode (tk ) amount of energy depending upon d(tk ). As a result, theenergy in the battery at the start of the tk+1 epoch is given by:
ebat t (tk + 1) = ebat t (tk ) + eharvest (tk ) − enode (tk )(1)
We assume no losses due to battery inefficiencies. This can be assumed without loss of generality
because any such losses can be lumped together as increase in node power consumption. The
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:6

S. Shresthamali et al.

distance from energy neutrality of the sensor node, or ENP, in a particular epoch tk , edist (tk ) is
given as follows:
edist (tk ) = eharvest (tk ) − enode (tk )(2)
In [9], the authors present mathematical guidelines to calculate the battery size and the required
battery level to account for the variance in the energy harvested by the system and consumed by
the load. We follow their formulation in to determine the size of the battery and the optimum initialbattery level required for ENO.
3.2

Reinforcement Learning

Reinforcement learning is a machine learning technique where the machine (or agent) learns
through experience rather than through instruction. The agent interacts with an environment andreceives feedback in terms of a scalar reward signal. Through trial and error, the agent learns which
actions are favorable depending upon which state it might find itself in. With enough experience,
we expect the agent to come up with an optimal policy that will maximize its long term cumulative
reward.
A general RL model consists of a finite state space S, an agent capable of executing a set ofactions A, and an environment that reacts to the said actions. The environment reacts to the agent’s
choice of action expressed by a scalar reward signal defined by the reward function R : S × A → R.
The action, a, to be executed when the agent is in a particular state, s, is dictated by the policy,
π = {(s, a)|a ∈ A, s ∈ S }. This is denoted by π (s) = a.
Each agent-environment interaction occupies an epoch. At epoch tk , the agent is in state sk ∈ S.
The agent takes an action, ak ∈ A, according to some policy π . The environment reacts to thisaction by changing the agent’s state to a new state sk+1 ∈ S and rewarding the agent with somescalar reward r k .
The objective of the agent is to find a policy that maximizes at each time step the expecteddiscounted sum of future reward. An optimal policy, π ∗ , maximizes this quantity for all state-action
pairs.
To give a measure of the the goodness of a particular action, a, according to some policy π , when
the agent is in some state s, we assign each state-action pair a Q-value [21]. The Q-value, Q π (s, a)is defined as the expected sum of discounted rewards starting from state s, taking action a andfollowing policy π thereafter. Mathematically it is expressed as," N −1#Õπk
Q (s, a) = E
γ r (sk , ak )
(3)
k =0

where s 0 = s, a 0 = a, ak = π (sk ). γ , 0 < γ < 1, is a discount factor. Equation 3 assumes thatthe agent-environment interaction lasts for N epochs. The Q-values for the optimal policy π ∗ isdenoted by Q ∗ . In this paper, we use SARSA(λ) algorithm to learn the Q-values. Determining the
optimal actions is trivial once Q ∗ is known. For each state s, the action a that maximizes Q ∗ (s, a) is
the optimal action. Choosing actions in this way is called a greedy policy.
3.2.1 SARSA:. SARSA stands for State-Action-Reward-State-Action. The Q-value, Q π (s, a), fora state-action pair (sk , ak ), corresponding to a policy π , is estimated by considering the agent’s
transition to another state-action pair (sk +1 , ak +1 ) and the reward, r k , it receives in the process.
The agent starts out in state sk , takes action ak according to some policy π . As a result, it receives
a reward r k and is transported to another state sk +1 . The agent then considers taking the next
action ak +1 according to the policy π . At this point, Q(sk , ak ) is updated as follows:
Q π (sk , ak ) ← (1 − α)Q π (sk , ak )+α [r k + γQ π (sk +1 , ak +1 )]ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

(4)

Adaptive Power Management in Solar Energy Harvesting Node using ...

39:7

where α, 0 < α < 1, is the learning factor.
We maintain a Q-table of all possible state-action pairs and their corresponding Q-values. Witha Q-table, finding the optimal action requires only a single lookup. This is not a computationallyintensive process once the Q-values have been sufficiently learned.
3.2.2 Eligibility Trace: When an agent passes through a series of states by performing a sequenceof actions and receives a reward at the end of an episode, assigning credit to the appropriate stateaction pairs becomes an issue. To resolve this, we introduce a memory variable for each state-action
pair called the eligibility trace. The eligibility trace for a state-action pair at epoch tk is denoted byek (s, a) ∈ R ≥0 . During each epoch, the eligibility trace for all state-action pairs decays by γ λ, andthe trace for the state-action pair visited on epoch tk is incremented by 1, i.e.
(γ λek −1 (s, a)if (s, a) , (sk , ak )ek (s, a) =(5)
γ λek −1 (s, a) + 1 if (s, a) = (sk , ak )for all (s, a), where λ, 0 < λ < 1, is a parameter that allows us to specify the strength with which
Q-values of early state-action pairs are updated as a consequence of the final reward. The valueof e(s, a) is a measure of how influential the state-action pair (s, a) was in obtaining the reward at
the end of an episode. We combine the concept of eligibility traces with SARSA learning to get
SARSA(λ). This is explained in more detail in the following section.
3.2.3 ϵ-greedy policy: We initialize the Q-table optimistically [21] by assigning all state-action
pairs high Q-values so that the algorithm is encouraged to explore more during the beginning of
the training period[21]. With sufficient training, the Q-values should converge to their true values.
SARSA learning converges to an optimal Q table when all state-action pairs have been visited
infinitely often. However, acting greedily before convergence may lead to sub-optimal policies
because the agent would not have had the opportunity to sample state-actions pairs that might
have led to higher returns. In order to avoid this, we follow an ϵ-greedy policy. This mean theactions are chosen greedily most of the time, but with probability ϵ, a random exploratory action is
selected.
4

SARSA FOR ENO

In this section, we explain how we set up the RL framework for our problem. We also present thealgorithm for SARSA(λ) learning.
4.1

RL Framework

The RL framework defines the state space and the action space. The environment, in this context,
consists of a stochastic energy source and the battery. It reacts to the agent by specifying a reward
based on the duty cycle chosen and a new state depending upon the amount of energy harvested,
the reserve battery level and the weather forecast information.
We train the agent in episodes. The agent takes a sequence of actions and traverses through aseries of states until the end of the episode (in our case an episode consists of 24 epochs). At the endof the episode, the agent is awarded some reward which it uses to evaluate its actions and upgrade
its policy (Q-table). The state definitions, action space and the reward scheme are described asfollows.
4.1.1 State Space. Given the statistics of the environment, the optimum battery level, B 0 , can be
determined by using the formulation in [9]. What this means is that if the agent starts out with
B 0 amount of battery at the start of every day, it is able to accommodate for the days with theleast energy harvested as well as days with the maximum energy harvested without depleting
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:8

S. Shresthamali et al.

the battery completely or allowing it to be overcharged. We use this battery level to calculate thedistance from energy neutrality. Ideally, we would like the agent to begin everyday with B 0 amountof battery and end with exactly B 0 remaining. If the battery level at epoch tk is B(tk ), we calculate
the distance from energy neutrality or ENP, edist (tk ), using B 0 as reference according to Equation
6.
edist (tk ) = B(tk ) − B 0(6)edist (tk ) is used to determine Sdist (tk ), the state of distance from energy neutrality. Using thisinformation for state definition makes our approach largely independent of the actual battery
capacity and thus more generally applicable. However, the agent also takes into account whether
the battery is in danger of being overcharged or completely depleted. This is reflected in Sbat t (tk ),the state of the battery reserve level. State Sday (tk ) gives an indication of what kind of weather theagent may expect. With this state information, the agent is able to use different strategies to achieve
ENO depending on how much energy it can expect to harvest. Finally, the state of the amountof energy being harvested during the present epoch is given by S eharvest (tk ). Hence the different
states in which the agent can exist is given by combination of Sbat t (tk ), Sdist (tk ), S eharvest (tk ), andSday (tk ) i.e.
(Sbat t (tk ), Sdist (tk ), S eharvest (tk ), Sday (tk )) ∈ S
4.1.2 Action Space. The action space, A, is defined as the set of discrete duty cycles that can bechosen. A ∈ (Dmin , Dmax ), where Dmin and Dmax are the minimum and maximum duty cycle ofthe sensor node. The agent chooses one duty cycle in each epoch.
4.1.3 Reward Function. The reward indicates what kind of behavior best serves our objective.
In our RL model, the reward awarded at the end of an episode depends simply on the distancefrom energy neutrality at the end of the episode. The "goodness" of any action in the middle ofan episode cannot be judged without taking the effect of other actions in the episode. Hence wewait until the end of the episode to judge the "goodness" of the sequence of actions chosen. This isalso a fair reward system considering that ENO-Max operation can be achieved by a number ofdifferent methods i.e. it is not always necessarily the case that there is only one unique optimal
policy. This is because we have not included battery inefficiencies in our formulation. As a result,the system is indifferent to using energy directly from the solar panel or from the battery.
Ideally we would like the agent to learn a policy to ensure zero deviation from its initial battery
level at the end of each episode. Low deviation are awarded large rewards and larger deviations areawarded with lower rewards.
4.2

SARSA(λ)

The reward at the end of an episode is used to update the Q-values of the state-action pairs according
to Algorithm 1 [21]. The use of eligibility traces allows us to propagate the reward backwards toall the state-action pairs that contributed to it. Before the training, the Q-table is optimisticallyinitialized to a high value qinit. At the start of each episode, the eligibility traces are reset and anaction is selected randomly. The agent then determines its state and uses an ϵ-greedy policy tointeract with its environment until the end of the episode. The reward it receives at the end of each
episode is used to update the Q-table towards a better estimate.
4.3

Training Setup

We use historical data for training the agent. We take solar data of a year and apply SARSA(λ)
learning for each day of that year for N number of iterations. The agent is trained in three phases.
In the first phase of training, the initial battery level is initialized to some middle value. In the
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

Adaptive Power Management in Solar Energy Harvesting Node using ...

39:9

ALGORITHM 1: SARSA(λ) algorithm
Initialize Q(s, a) = qinit and e(s, a) = 0 for all s, a;for each episode do
Initialize s, a;for each step of the episode doTake action a, observe reward r and next state s 0 ;Choose next action a 0 from Q using ϵ-greedy policy;δ ← r + γQ(s 0, a 0 ) − Q(s, a);e(s, a) ← e(s, a) + 1;
for all (s, a) do
Q(s, a) ← Q(s, a) + αδe(s, a);e(s, a) = γ λe(s, a);ends ← s 0; a ← a 0;
end
end

second and third phase of training, this is initialized to high and low values respectively. This isdone so that the agent is able to ensure ENO even when battery levels are not initialized at optimallevels. We observe the behavior of a trained agent by simulating it in an environment which it has
not experienced during training. We allow the agent to learn as it interacts with the environment
so that it can adapt and re-calibrate itself if need be.
5

SIMULATION METHODOLOGY

Here we describe the specifications of our system model and the parameters that are used during
training and implementation.
5.1

System Setup

We base our system specification on a scaled up version up of a TMote Sky node powered by a 3.6V,
2200 mAh NiMh battery and a 6 W solar panel (220mm × 175mm). The node power consumption
varies from approximately 100 mW to 20 mW depending upon whether the node is sensing,transmitting or receiving data. We scale up these values roughly by a factor of five. We chose thisparticular configuration simply for the sake of example. Any change in this system (for e.g. change
in battery capacity or node power consumption) is accommodated for by the power manager due
to its adaptive nature. This in fact is the essence of our approach - our power management policy
is fluid enough to find a near-optimal policy for any realistic system specification without the needfor any intervention by the user.
5.1.1 Energy Source: To simulate the solar energy harvested, we acquire global solar radiation
data from the Japanese Meteorological Agency (JMA) website http://www.jma.go.jp. JMA provides
hourly data on the global solar radiation several locations in Japan. As a result, we fix an epoch to
be an hour long for our purpose. We use the solar radiation data to calculate the electrical energy
generated by a solar panel.
5.1.2 Sensor Node: We consider a general sensor node that consumes energy varying from 100
mWh to 500 mWh during each epoch according to the specified duty cycle. We assume the power
consumption remains constant within each epoch. The possible duty cycles are 20%, 40%, 60%, 80%
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:10S. Shresthamali et al.

and 100%. We do not consider sensing latency inherent in sensor nodes for sake of simplicity. The
power manager is indifferent to how the node allocates its power consumption.
5.1.3 Battery: Using the guidelines in [9], we calculate the battery size, B MAX , and the optimal
initial battery level, B 0 . Using statistics for the year 2010, we arrive at B MAX = 40000mW h and
B 0 = 60% of B M AX . We assume an ideal battery for the sake of simplicity. Any inefficiencies of thenode, solar cell and the battery can be lumped together as an increase in node power consumption
in this context.
5.2

SARSA(λ) Parameters5.2.1 Action Space. The action space, A, defines the actions that can be chosen by the agent. In
our case, the action space is A = {20%, 40%, 60%, 80%, 100%}.
5.2.2

State Definitions. The state of the system at some epoch tk is given by
Sk = (Sbat t (tk ), Sdist (tk ), S eharvest (tk ), Sday (tk )).

The state of the agent is determined from the actual values observed by the agent. Since thesevalues are continuous in nature, it is necessary to discretize and define which values correspond towhich states. We use the following to assign states from actual observed values.
Sbat t (tk ) ∈ {Sb1 , Sb2 , Sb3 } gives information about whether the agent is in danger of depletingits battery or overcharging it. This is determined by the value of ebat t (tk ) according to Table 1.
Table 1. Sbat t (tk ) Assignment

Sbat t (tk )
Range
Sb1
ebat t (tk ) < 20% of B MAX
Sb2
20% of B M AX ≤ ebat t (tk ) < 80% of B MAX
Sb3
80% of B M AX ≤ ebat t (tk ) < 100% of B MAX
Sdist (tk ) ∈ {Sd 1 , Sd 2 , ..Sd 40 } gives information on how far the agent’s operation is from ENO.
This state is determined by the value of edist (tk ) according to the following.
• Sdist (tk ) ∈ {Sd 22 , ..Sd 40 } correspond to states in which edist (tk ) < 0 i.e. the battery is at alevel lower than B 0 .
• Sdist (tk ) ∈ {Sd 21 } corresponds to the state of perfect energy neutrality i.e. edist (tk ) =B 0 − ebat t (tk ) = 0;
• Sdist (tk ) ∈ {Sd 1 , Sd 2 , ..Sd 20 }correspond to states in which edist (tk ) > 0 i.e. the battery is at
a level higher than B 0 .
Neighboring states are spaced at a distance of 1000 mWh from each other.
S eharvest (tk ) ∈ {S e1 , S e2 ...S e7 } gives information about how much energy is being harvested at
epoch (tk ). Its value is determined by eharvest (tk ) according to Table 2.
Sday (tk ) ∈ {S f 1 , S f 2 ...S f 6 } gives the agent general information about what kind of weather it
can expect during the day as shown in Table 3. In real world application, this information canbe obtained from weather apps or websites. For our case, we differentiate each day into one ofsix different types according to the value of eday , the total amount of energy harvested in thatparticular day. Since we are training on historical information, eday can be easily calculated prior
to the training by using Equation 7.
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

Adaptive Power Management in Solar Energy Harvesting Node using ...

39:11

Table 2. S eharvest (tk ) Assignment

S eharvest
S e1
S e2
S e3
S e4
S e5
S e6
S e7

Range
eharvest (tk ) = 0mW h
0mW h < eharvest (tk ) ≤ 100mW h
100mW h < eharvest (tk ) ≤ 500mW h
500mW h < eharvest (tk ) ≤ 1000mW h
1000mW h < eharvest (tk ) ≤ 1500mW h
1500mW h < eharvest (tk ) ≤ 2000mW h
eharvest (tk ) > 2000mW h

eday =

24
Õ

eharvest (tk )

(7)

k =1

Table 3. Sday (tk ) Assignment

Sday
Sf 1
Sf 2
Sf 3
Sf 4
Sf 5
Sf 6

Weather
Range
Very little sun
eday (tk ) < 2500mW h
Overcast
2500mW h ≤ eharvest (tk ) < 5000mW h
Partly Cloudy
5000mW h ≤ eharvest (tk ) < 8000mW h
Fair
8000mW h ≤ eharvest (tk ) < 10000mW h
Sunny10000mW h ≤ eharvest (tk ) < 12000mW h
Very Sunnyeharvest (tk ) ≥ 12000mW h

500Reward

300
100
0

-100
-300
-500

-0.8

0
-0.4
0.4
edist(T), Distance from energy

0.8

x104

neutrality (mWh)

Fig. 2. Reward Function

5.2.3 Reward Function. The reward function in Figure 2 shows the relationship between the
reward and the distance from energy neutrality at the end of the episode (day), edist (T ). Since
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:12

S. Shresthamali et al.

edist (T ) cannot always be exactly zero, we consider a margin of ±1000 mWh deviation as acceptable.
The symmetry of the reward scheme is due to the fact that both positive and negative deviations
from ENO have the same effect on the reward. During training, the rewards oscillate around some
final value which introduces some noise in the reward. The sharp profile of the reward function
allows the learning algorithm to arrive at true Q-values in spite of this noise. The gradual slope
between ±1000 mWh and±5000 mWh deviation enables the agent to incrementally improve its
performance without causing unstable oscillations during learning. This reward function is a novel
contribution of our work. It is simple and intuitive and best reflects the ENO objective. Moreover,this reward function is independent of system model specifications and thus has a general scope of
application.
5.3

Training Parameters

We execute Algorithm 1 with α = 0.001, γ = 0.8 and λ = 0.3. As with most RL problems, thesevalues were determined empirically rather than through mathematical methods. We evaluated thesystem with different combinations of the values of the hyper parameters and chose the combinationthat performed the best. The system is somewhat sensitive to the values of α and ϵ. Using a highvalues for α (learning rate) and ϵ causes large oscillations in Q-values during training. Hence, wechose smaller values but compensated with a larger number of iterations during learning.
Each epoch is an hour long with one episode consisting of 24 epochs. During training, the agent
iterates N (N = 106 ) times for each day of the year. For the three phases of training, we fix theinitial battery level B I N IT at 60%, 80% and 20% of B MAX .
We train our agent with weather data of Tokyo for the year 2010. We then observe its performancefor the year 2011 in Tokyo and Wakkanai. Wakkanai lies in far north of Japan and experiences
a drastically different weather than that of Tokyo. We use this change in location and climate toobserve the node’s adaptive behavior.
5.4

Evaluation Metrics

We refer to our power management policy as SARSA Policy. We compare our policy to a power
management strategy mentioned in [9] referred here as Offline Policy. Offline Policy uses linear
programming optimization methods with non-causal data on energy harvesting opportunities todetermine the optimal duty cycles. The results of this method are presented here as an estimateof the upper limit of performance. This is not a practical method of power management because
non-causal data on energy harvesting opportunities is not available in real life. The Offline Policy
solutions have real continuous values and therefore to ensure fair comparison, the values of the
duty cycles are rounded off to the nearest possible duty cycle of the system. As a result of thisrounding off process, the Offline Policy also rarely every achieves perfect ENO. The Offline Policy
uses an optimization window of one day (24 hours) to calculate the duty cycles. This also ensuresfair comparison with our method because our SARSA(λ) agent is also trained in one-day episodes.
We express the battery levels, duty cycles and energy harvested in percentage of their maximum
values. ENP are expressed as percentages of the maximum battery value B MAX . We use root meansquared (RMS) values of ENP to compare performance between different policies.
6
6.1

RESULTS
Learning Convergence

Figure 3 shows the convergence of SARSA(λ) to its final optimal policy as the agent trains overa number of iterations. The graph shows the end of the day deviations from the initial optimalbattery level, B 0 for each iteration during training on days 58 and 69 of the year 2010 (Tokyo).
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

Adaptive Power Management in Solar Energy Harvesting Node using ...

Day 695000

Absolute Distance from Energy Neutrality
(mWh)

39:13

4000
3000
2000
1000
0

α = 0.001
ε = 0.001

Day 583000

γ = 0.8
λ = 0.3

2000
1000
0

0

1

2

3

4

5

6

7

Iterations

8

9

10

x104

Fig. 3. Policy Convergence

We see that the agent is learning vigorously during the first 30000 to 40000 iterations. After
50000 iterations, it has figured out the optimal policy as is evidenced by the low deviation fromenergy neutrality in the figure. For the next 50000 iterations, it follows this policy greedily taking
an occasional random action to see if it might lead to a better policy.
The fluctuations in the deviation from energy neutrality is the consequence of the limited numberof possible discrete power consumption modes in the node. Since the harvested energy can takecontinuous values, it is highly unlikely that perfect energy neutral operation will ever be achieved.
There will always be a small amount of error. We allow an error margin of ±1000 mWh (±2.5% of
B MAX ) for the best case scenario. We tolerate up to ±5000 mWh (±12.5% of B MAX ) of deviation.
This is also reflected in the reward function shown in Figure 2. Also, a small change in one of thenode’s actions may cause the deviation to jump between positive and negative values. This may
explain the oscillation in deviation values as the agent converges to an optimal policy.
6.2

Energy Neutral Operation

Figure 4 shows the comparison of SARSA Policy with the Offline Policy for January 29, 2011 Tokyo.
The Constant Duty Cycle Policy is determined by simply dividing the total energy harvested by
24 i.e. the total number of hours in one day. Of course, this constant duty cycle is not a realistic
duty cycle that is achievable by the agent and is shown for comparison purposes only. The green
dotted-dashed line indicates the initial level of the battery (B 0 ).
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:14

S. Shresthamali et al.
Duty Cycle(Constant Duty Cycle Policy)

Duty Cycle(SARSA Policy)

Duty Cycle(Offline Policy)

Battery (%)(SARSA Policy)

Battery (%)(Offline Policy)

Percentage (%)

Duty Cycle (%)

100
80
60
40
20
Initial Battery
Level, B080
60

RMS ENP40
20
0Energy Harvested (%)

SARSA: 5.03%
Offline: 5.06%

DAY 29
4

8

12

16

20

24

Time (Hour)Fig. 4. SARSA and Offline Policy

It is worth noting that although the duty cycles of the Offline Policy and SARSA Policy differ
from each other, they both have very similar performance. Both start the day off with 60% battery
end with near about the same battery level. The ENP at the end of the day for SARSA Policy andOffline Policy was 491.875 mWh and 191.875 mWh respectively. SARSA Policy violated the energy
neutrality by only 300 mWh (0.75% of B M AX ) more than that of Offline Policy. In fact, SARSA Policy
shows slightly lesser RMS deviation (5.03%) as compared to the Offline Policy (5.06%). This showsthat SARSA policy comes very close to optimal performance using only a general knowledge of theweather forecast.
As mentioned before, the Offline Policy optimizes its performance when considering one day
at a time. In Figure 5, we compare SARSA Policy with the Offline Policy with a one-day window(shown in violet) as well as with an Offline Policy with a 30-day window (shown in blue) for 2011,
Tokyo. We allow the duty cycles of the Offline Policy with a 30-day window to have continuous
values between 10% and 100% and as a result, perfect energy neutrality is achieved using this policy.
We observe the battery profiles for different polices for a 30-day period. We observe that bothSARSA Policy and one-day window Offline Policy have similar behavior and try and maintain thebattery level at 60% of B M AX at the end of every day. In contrast, the Offline Policy with a 30-day
window deviates significantly from the optimal battery level during the middle of the 30-day period.
The SARSA Policy can also be modified to optimize for a longer window. Such longer windows
optimization techniques show better performance especially when the battery is nearing its limits.
However they require more computation and exponentially longer training sessions.
Figure 5 also shows the battery profiles for two other policies - the constant duty cycle policy(shown in dotted blue) and a naïve battery-centric policy (shown in dashed orange). By averaging
the non-causal information about the energy harvested over the 30 day period, we can determine
the constant duty cycle to ensure ENO. This policy is not adaptive at all. Using a constant duty
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

Adaptive Power Management in Solar Energy Harvesting Node using ...
Battery isovercharged

100

39:15

SARSA Policy
Offline Policy

Battery (%)

80
Offline Policy(30 day window)
60

Constant
Duty Cycle

40

Naive Policy

20

EnergyHarvested (%)

0
278

282

287

292

297

302

307

Time (Day)
Fig. 5. Comparison of different policies

RMS Deviation from Energy Neutrality
23.82%

19.44%

10.47%

Constant
Duty Cycle

Offline(30 day
horizon)

5.27%
3.46%

Offline

SARSA

Naïve
(BatteryCentric)

Fig. 6. Energy neutral performance of different policies

cycle policy may lead to battery overcharge/depletion and therefore wastage of energy and node
failure. We can observe that some amount of energy is wasted due to overcharging during Day 286
as a result of this policy. The Naïve policy is the simplest adaptive policy. It is battery-centric inthat the duty cycle is proportional to the battery reserve level. Higher battery drives the node withhigher duty cycles and vice versa. While this policy is simple to implement, it is not very intelligent.
For instance, on Day 288, the Naïve policy deviates quite far from the optimal battery level anddrops to almost 20% on Day 292. If the days following Day 292 were not sunny enough, the battery
could very well have been depleted. Figure 6 shows the RMS deviation from energy neutrality atthe end of the 30 day period. We can see that the Naïve Policy suffers from the largest deviation
at more than 23%. SARSA Policy and one-day window Offline Policy show very little deviation
3.46% and 5.27%. The constant duty cycle and 30-day horizon Offline policy exhibit perfect energy
neutrality. However this comes at the cost of higher deviations as illustrated in Figure 5.
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:16

S. Shresthamali et al.

70

SARSA Policy without Weather Forecast

Overall lower deviation from B0 when
including weather forecast information

50
30

10
0

308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350

Percentage (%)

SARSA Policy with Weather Forecast

Time (Day)
Fig. 7. Effect of weather forecast information

6.3

Effect of including weather forecast

In Figure 7 we compare the behavior of SARSA policies with and without forecast informationfrom Day 308 to 324 (Wakkanai 2011). The policy that does not involve weather forecast in its
decision making (violet) deviates more from the optimal battery level (shown in dotted-dashedgreen) than the policy that considers weather forecast information (shown in red). Both policies
strive to achieve energy neutrality but the policy with weather forecast is more successful because
it leverages on the weather forecast information to make better decisions. On the other hand,
the weather agnostic policy uses a general policy for all weather types and this leads to lower
performance in comparison.
6.4

Adaptation to seasonal changes

Figure 8 and 9 compare the SARSA Policy and Offline Policy for a week in spring and winter ofTokyo 2011. Figure 8 shows the plots for the week starting from February 27. The second and third
day receive very little sunshine. However, starting from the fourth day, it gets a lot sunnier. We
can see how SARSA adapts its strategy when it is anticipating a sunny or non-sunny day. We also
observe that for the last three days of the week, both SARSA Policy and Offline Policy max out the
duty cycle to 100% to be able to use all of the energy being harvested. In spite of their best efforts,
the amount of energy that is harvested exceeds what can be consumed. This surplus is stored in
the battery and is reflected in the rising battery level.
Conversely during the week (starting from Nov 29) in winter (Figure 9), the amount of energy
harvested is not sufficient to sustain even the minimum duty cycle. During the first three days,
both policies try to maximize their duty cycle as much as possible. However from the fourth day
onwards, both policies fall back to the lowest duty cycle. The last day of the week is quite sunnyand so both policies replenish their battery back to the optimum level. We see that SARSA has asimilar RMS ENP compared to the Offline Policy during both spring and summer. This shows that
the SARSA Policy is able to maintain near-optimal performance while still being able to adapt toseasonal changes.
6.5

Adaptation to climatic changes

To simulate a change in environment due to change in location, we observe the behavior of the nodein Wakkanai, a place with a climate drastically different from which the node was originally trainedon. Figure 10 shows the performance of both SARSA Policy and Offline Policy for a two-weekperiod starting from February 4. We can see that both policies have similar behavior.
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

Adaptive Power Management in Solar Energy Harvesting Node using ...
Duty Cycle(SARSA Policy)

Duty Cycle (%)

10080

Duty Cycle(Offline Policy)

60

RMS ENP40SARSA:
Offline:Percentage (%)

20

6.71 %
6.73 %

Distance fromenergy neutrality (%)(SARSA Policy)

EnergyHarvested (%)

20
0

58

Battery (%)(SARSA Policy)
Battery (%)(Offline Policy)

60
40

39:17

59

60

61

62

63

64

Distance fromenergy neutrality (%)(Offline Policy)
65

Time (Day)
Fig. 8. Tokyo Spring 2011

Percentage (%)

Duty Cycle (%)

Duty Cycle(SARSA Policy)
RMS ENP100

Duty Cycle(Offline Policy)

SARSA: 6.80 %Offline: 5.08 %80

60

Battery (%)(SARSA Policy)

40
20

Battery (%)(Offline Policy)

60

Distance fromenergy neutrality (%)(SARSA Policy)

EnergyHarvested (%)

40
20
0
332

333

334

335

336

337

338

Distance fromenergy neutrality (%)
339 (Offline Policy)

Time (Day)
Fig. 9. Tokyo Winter 2011

The duty cycles corresponding to SARSA Policy have more variance than that of Offline Policy.
This may be due to the fact that Offline policy has perfect prior knowledge of all energy harvestingdata and so can optimize its policy better. SARSA Policy on the other hand does not have thisinformation and instead has to decide on an action only after it observes the current harvested
energy state. This may also explain why SARSA Policy has higher RMS ENP than Offline Policy.
We believe that SARSA Policy adapts quite well to Wakkanai environment because we have useddistance from the energy neutrality as one of the state definition. Since the SARSA Policy is largely
independent of the battery level (except when it is near the extreme limits), the strategy it uses
during severe Tokyo winters also works for Wakkanai. Figure 11 is a color map that shows thedeviation from energy neutrality for both policies in the year 2011. The top half shows the resultsfor Tokyo whereas the bottom half is for Wakkanai. Each of the four color maps consists of 12 rows
corresponding to each month in the year 2011. Each cell of a month row corresponds to a day in
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:18

S. Shresthamali et al.
Duty Cycle(SARSA Policy)

Duty Cycle (%)

80

Percentage (%)

100

60

Duty Cycle(Offline Policy)

RMS ENPSARSA: 7.01%
Offline: 3.14%

Battery (%)(SARSA Policy)

60
40

Battery (%)(Offline Policy)

20

Distance fromenergy neutrality (%)(SARSA Policy)

40
Distance fromenergy neutrality (%)(Offline Policy)

20
0

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

Time (Day)

EnergyHarvested (%)

Fig. 10. Wakkanai Feb 2011

the month and its color indicates the deviation from energy neutrality at the end of that day. Bluecells indicate positive deviation i.e. more energy is harvested than that is consumed. Red indicatesthat more energy is consumed than that was harvested. Greener the cell, the lesser the deviation.
The RMS ENP for each case is also shown alongside the figures.
The battery is initialized to the optimal value at the start of each day to get a fair idea of deviationfrom energy neutrality for each day. SARSA Policy is able to achieve ENO-Max operation in mostcases. The RMS ENP for Offline Policy gives us an estimate of the best possible performance. We
observe that SARSA Policy comes very close to achieving optimal performance by adapting to thechanges in the environment.

OfflineRMS ENP: 6.61% Policy

RMS ENP: 5.70%

Day

RMS ENP: 7.17% Day

RMS ENP: 5.94%

Tokyo

SARSA
PolicyWakkanaiMonth< -5000 mWh

Distance from Energy Neutrality> 5000 mWh

Fig. 11. Wakkanai vs Tokyo

ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

Adaptive Power Management in Solar Energy Harvesting Node using ...
6.6

39:19

Adaptation to battery degradation

To simulate the degradation of battery of the sensor node, we observe the behavior of SARSA Policy
for 2011 Wakkanai weather with only half the original battery capacity. B MAX is now reduced
to 20000 mWh and B 0 = 60%o f B M AX becomes 12000. The SARSA Policy now tries to achieve
ENO around this new B 0 . Since the SARSA Policy gives greater weight to the distance of energy
neutrality than the actual battery level, the policy is able to achieve energy neutral performanceeven when the battery capacity is halved. This is shown in Figure 12. The RMS ENP for Offline
Policy again gives us an estimate of the upper limit in performance. We see that our policy comesvery close to achieving this in spite of a drastic change in device parameters. When comparing with
the Offline Policy, we see that SARSA Policy show a slightly reduced performance (as evidenced by
fewer green cells) but has an overall satisfactory result.

Offline Policy
MonthWakkanaiSARSA PolicyDay

RMS ENP: 10.75% Day< -5000 mWh

RMS ENP: 11.55%

Distance from Energy Neutrality> 5000 mWh

Fig. 12. Performance with half battery capacity

6.7

Adaptation to changes in device parameters

In real life application scenarios, the sensor node may have to accommodate for changes in its
device parameters such as a decrease in energy harvesting efficiency and energy efficiency of thesensor node. Our proposed solution is able to adapt to such changes and learn to perform optimallyin such scenarios. To simulate such a setting, we observe how SARSA policy learns to adapt toconditions when we• halve the solar cell output• increase the node’s power consumption by 2.5 times (to simulate a degradation in thenode’s energy efficiency)
The sensor node (agent) has to be put in a "learning mode" so that it can adapt to these changes
in working parameters. In this learning mode, the agent is allowed to train for 1000 iterations with
α = 0.1 and ϵ = 70 in its new environment. The high learning rate and exploration rate allows it toquickly make changes in its learned Q-table. An additional advantage of this adaptive behavior isthat the sensor node can make up for any initial calibration errors by the user. Even if the sensor
node is assumed to be deployed in an environment it was not initially designed for, it can quickly
adapt and work optimally. Thus, even though the simulation and real world applications may differ,
our approach to power management control is able to accommodate these differences and still
perform optimally.
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:20S. Shresthamali et al.

Figure 13 shows the battery profiles for SARSA and Offline Policy with default device settings(without being put into the learning mode) for October 31, 2011. This is similar to the results in
Figure 4. The only difference is that it is for a different date. We use this as the baseline performance.
October 31, 2011

SARSA Policy

100

Duty Cycle (%) 80

80

Battery (%)

60

60

ENP (%)RMS ENP: 5.17 %40

EnergyHarvested (%)

ENP = 240 mWh

RMS ENP: 5.31 %40
ENP = 40.63 mWh
20

20

0

Offline Policy

100

1

4

8

12

16

20

0

24

1

4

8

12

16

20

24

Time (Hour)

Time (Hour)Fig. 13. Default Device Settings

Next, we observe the performance of the SARSA Policy when the solar panel output is reducedby half. This can be due to decrease of solar panel efficiency or a mismatch in design parameters
during testing and implementation. The battery profiles for SARSA and Offline Policy is shown in
Fig 14. We see that SARSA is able to achieve node level energy neutrality with very little battery
deviation at the end of the day. In fact, in this particular case, SARSA policy has a lesser deviationfrom ENO than Offline Policy. Although the Offline Policy does represent the theoretical upper
limit, the rounding off process involved can introduce some errors such as in this case.
October 31, 2011

SARSA Policy

100

Percentage (%)

Offline Policy

100

RMS ENP: 1.62 %80

Duty Cycle (%) 80

60

Battery (%)

RMS ENP: 3.03 %60

ENP (%)
40

EnergyHarvested (%)

200

ENP = 70.31 mWh

1

4

8

12

16

Time (Hour)

20

24

40

20

0

ENP = 270.31 mWh

1

4

8

12

16

20

24

Time (Hour)Fig. 14. Performance with half solar panel capacity

Finally, we consider the case when the sensor node consumes more power than it was initiallyassumed. This again maybe due to decrease in the node’s working efficiency or mismatch in design
and implementation. In Figure 15, we see that SARSA is able to achieve better performance than
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

Adaptive Power Management in Solar Energy Harvesting Node using ...

39:21

Offline Policy in this case also. SARSA is achieves an ENP as low as 70.31 mWh at the end of theday.
October 31, 2011

SARSA Policy

100

Percentage (%)

Offline Policy

100

RMS ENP: 5.12 %RMS ENP: 1.62 %Duty Cycle (%) 80

80

Battery (%)

60

60

ENP (%)
40

EnergyHarvested (%)

200

ENP = 70.31 mWh

1

4

8

12

16

Time (Hour)

20

24

40

20

0

ENP = 390.63 mWh

1

4

8

12

16

20

24

Time (Hour)Fig. 15. Performance with increased node power consumption

7

CONCLUSION

We observe that our SARSA(λ) based RL approach achieves near perfect ENO making autonomous
operation a possibility. We also see that our definitions of the state results in a highly adaptive
behavior. Our proposed method is able to adapt to changes in weather, location (climate), battery
degradation and device parameters which makes the sensor node robust in its operation. In addition,
our state definition and general reward formulation scheme allows for general application of ourpower management method independent of the system it is being implemented in. We also showthat inclusion of weather forecast information enhances the performance of the proposed scheme.
ACKNOWLEDGMENTS
This work was partially supported by JSPS KAKENHI Grant Number 16K12405. The first author
acknowledges the Japanese Government (MEXT) Scholarship for his study in The University ofTokyo.
REFERENCES[1] Luca Benini, Giuliano Castelli, Alberto Macii, and Riccardo Scarsi. 2001. Battery-driven dynamic power management.
IEEE Design & Test of Computers 18, 2 (2001), 53–60.
[2] Pol Blasco and others. 2013. A learning theoretic approach to energy harvesting communication system optimization.
IEEE Tr. on Wireless Communications 12, 4 (2013), 1872–1882.
[3] Wai Hong Ronald Chan and others. 2015. Adaptive duty cycling in sensor networks with energy harvesting using
continuous-time Markov chain and fluid models. IEEE Journal on Selected Areas in Communications 33, 12 (2015),
2687–2700.
[4] Deniz Gunduz, Kostas Stamatiou, Nicolo Michelusi, and Michele Zorzi. 2014. Designing intelligent energy harvesting
communication systems. IEEE Communications Magazine 52, 1 (2014), 210–216.
[5] Jason Hsu and others. 2006. Adaptive duty cycling for energy harvesting systems. In Proc. of the 2006 ISLPED. 180–185.
[6] Roy Chaoming Hsu and others. 2014. A Reinforcement Learning-Based ToD Provisioning Dynamic Power Management
for Sustainable Operation of Energy Harvesting Wireless Sensor Node. IEEE Tr. on Emerging Topics in Computing 2, 2(2014), 181–191.
[7] Roy Chaoming Hsu and others. 2015. Dynamic energy management of energy harvesting wireless sensor nodes using
fuzzy inference system with reinforcement learning. In IEEE 13th INDIN. 116–120.
ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.

39:22

S. Shresthamali et al.

[8] Xiaofan Jiang, Joseph Polastre, and David Culler. 2005. Perpetual environmentally powered sensor networks. In
Proceedings of the 4th international symposium on Information processing in sensor networks. IEEE Press, 65.
[9] Aman Kansal and others. 2007. Power management in energy harvesting sensor networks. ACM Tr. on Embedded
Computing Systems 6, 4 (2007), 32.
[10] Aman Kansal, Dunny Potter, and Mani B Srivastava. 2004. Performance aware tasking for environmentally powered
sensor networks. ACM SIGMETRICS Performance Evaluation Review 32, 1 (2004), 223–234.
[11] Junaid Ahmed Khan and others. 2015. Energy management in wireless sensor networks: a survey. Computers &Electrical Engineering 41 (2015), 159–176.
[12] Cheng-Ting Liu and Roy Chaoming Hsu. 2011. Dynamic power management utilizing reinforcement learning withfuzzy reward for energy harvesting wireless sensor nodes. In 37th Annual Conference on IEEE Industrial Electronics
Society. 2365–2369.
[13] S Danish Maqbool and others. 2011. Analysis of adaptability of Reinforcement Learning approach. In IEEE 14th INMIC.
45–49.
[14] Nicolò Michelusi and others. 2013. Energy management policies for harvesting-based wireless sensor devices withbattery degradation. IEEE Tr. on Communications 61, 12 (2013), 4934–4947.
[15] Andrea Ortiz and others. 2016. Reinforcement learning for energy harvesting point-to-point communications. In 2016
IEEE International Conference on Communications. 1–6.
[16] Vijay Raghunathan, Aman Kansal, Jason Hsu, Jonathan Friedman, and Mani Srivastava. 2005. Design considerations forsolar energy harvesting wireless embedded systems. In Proceedings of the 4th international symposium on Information
processing in sensor networks. IEEE Press, 64.
[17] Luigi Rucco and others. 2013. A bird’s eye view on reinforcement learning approaches for power management in
WSNs. In 6th WMNC. 1–8.
[18] Navin Sharma and others. 2010. Cloudy computing: Leveraging weather forecasts in energy harvesting sensor systems.
In 7th IEEE SECON. 1–9.
[19] Sujesha Sudevalayam and Purushottam Kulkarni. 2011. Energy harvesting sensor nodes: Survey and implications.
IEEE Communications Surveys & Tutorials 13, 3 (2011), 443–461.
[20] Srikanth Sundaresan and others. 2009. Event-driven adaptive duty-cycling in sensor networks. International Journalof Sensor Networks 6, 2 (2009), 89–100.
[21] Richard S Sutton and others. 1992. Reinforcement learning is direct adaptive optimal control. IEEE Control Systems 12,
2 (1992), 19–22.
[22] Emre Ünsal, Taner Akkan, L Özlem Akkan, and Yalçın Çebi. 2016. Power management for Wireless Sensor Networks inunderground mining. In Signal Processing and Communication Application Conference (SIU), 2016 24th. IEEE, 1053–1056.
[23] Christopher M Vigorito and others. 2007. Adaptive control of duty cycling in energy-harvesting wireless sensor
networks. In 4th IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks.
21–30.

Received April 2017; revised June 2017; accepted July 2017

ACM Transactions on Embedded Computing Systems, Vol. 17, No. 4, Article 39. Publication date: October 2017.