Multi-objective Reinforcement Learning for Energy
Harvesting Wireless Sensor Nodes
Shaswot Shresthamali

Masaaki Kondo

Hiroshi Nakamura

Keio University
shaswot@acsl.ics.keio.ac.jp

Keio University
kondo@acsl.ics.keio.ac.jp

The University of Tokyo
nakamura@hal.ipc.i.u-tokyo.ac.jp

Abstract—Modern Energy Harvesting Wireless Sensor Nodes(EHWSNs) need to intelligently allocate their limited and unreliable energy budget among multiple tasks to ensure long-term
uninterrupted operation. Traditional solutions are ill-equipped todeal with multiple objectives and execute a posteriori tradeoffs.
We propose a general Multi-objective Reinforcement Learning(MORL) framework for Energy Neutral Operation (ENO) of
EHWSNs. Our proposed framework consists of a novel Multiobjective Markov Decision Process (MOMDP) formulation andtwo novel MORL algorithms. Using our framework, EHWSNs
can learn policies to maximize multiple task-objectives andperform dynamic runtime tradeoffs. The high computation andlearning costs, usually associated with powerful MORL algorithms, can be avoided by using our comparatively less resourceintensive MORL algorithms. We evaluate our framework on a
general single-task and dual-task EHWSN system model throughsimulations and show that our MORL algorithms can successfully
tradeoff between multiple objectives at runtime.

I. I NTRODUCTION
Energy Harvesting Wireless Sensor Nodes (EHWSNs) arebecoming increasingly popular as edge devices for the Internetof Things (IoT). They are capable of untethered, autonomousand long-term operation. Modern EHWSNs have sophisticatedSystem-on-Chips (SoCs) that enables them to perform avariety of tasks like sensing, communication and processing[1]. However, these nodes operate with very stringent energy
budgets and unpredictable energy availability. It is therefore
necessary for them to judiciously budget their energy among
multiple tasks to i) ensure energy neutrality for long-term
operation and ii) maximize the overall node utility. Energy
Neutral Operation (ENO) [2] requires that the average supply
and consumption of energy remain balanced for long-term
operation. On the other hand, maximizing node utility requires
intelligently expending energy for various tasks in order of
their user-defined priorities. Thus, optimal operation of ageneral EHWSN requires i) proportioning energy between different tasks and ii) ensuring that the gross energy consumption
does not violate energy neutrality.
Traditionally EHWSNs optimized for a single objective using low-compute analytic methods such as linear programming[2] and control systems [3], [4]. However, these solutions
were minimally adaptive and required significant hand-tuning.
As EHWSNs became more powerful, Reinforcement Learning(RL)-based methods were proposed as an alternative [5], [6].
In this paradigm, the nodes learn the energy management
policy on the fly as opposed to having pre-programmed

heuristics. During RL, the node (or agent) interacts with its
environment and explores various energy management policies
via trial-and-error. A reward function gives feedback to thenode which it uses to optimize its policies and maximize
future rewards. With RL-based policies, the nodes can adapt
to their specific working environment and therefore be used
in a deploy-and-forget fashion. Moreover, since they require
minimal application-specific design bias, they could be massively scaled to a large number of applications with little orno changes to the RL framework. This scalability of RL is anextremely important advantage over analytic methods because
it allows a single framework to work for potentially billions of
EHWSNs in diverse applications. However, these RL solutions
focus on only one aspect of the EHWSN operation such as
ensuring ENO [6] or maximizing throughput [7], [8] or thesensing rate [5], [9] and therefore are not suited for working
with multiple objectives.
Motivation: Modern EHWSNs require a general, integrated
solution that optimizes for multiple task objectives in addition
to maintaining ENO. Since all tasks draw energy from thesame limited source, tradeoffs need to be made based onuser-defined preference or priority. Generally, the relative
priorities are not known beforehand and can be adjusted only
after observing the possible tradeoffs. Hence, a multi-objective
optimization approach may be more suitable for this kind ofproblem.
Current multi-objective solutions for EHWSNs have proposed to reduce the multiple objectives to a set of constraints and a single objective [10]. They require very strong
application-specific assumptions and design-bias (e.g., noncausal information/ network dependent optimization objectives). This approach is not general and cannot scale well.
Traditional single-objective RL (SORL) solutions are also notsuitable for optimizing over multiple objectives. SORL isbased on learning from a single scalar feedback reward signal
corresponding to a single objective. They cannot accommodate
multiple sources of rewards (i.e., multiple optimization objectives). Some have tried to work around this by projecting
the rewards onto a scalar [5], [8], [11], [12]. However, as we
will see later, this approach requires strong prior assumptions,produces sub-optimal policies and cannot tradeoff dynamicallyat runtime. Thus current analytic and RL solutions are insufficient for optimization of modern EHWSNs.
Contributions: In this work, we propose a novel MORL

framework for multi-objective optimization of EHWSNs. We
use RL over analytic approaches due to its adaptivity andgenerality. Our proposed framework consists of a novel MultiObjective Markov Decision Process (MOMDP) formulation of
the EHWSN optimization problem and two MORL algorithms.
With our proposed solution, EHWSNs can learn intelligent
policies (with reduced learning costs) to i) maximize multiple
task objectives, ii) optimize gross device energy consumption
for long-term ENO and iii) dynamically tradeoff between
objectives during runtime. Specifically, we make the following
contributions in this paper:• We present a simple but general system model for multitask EHWSNs in Section III that accommodates multiple
tasks objectives and their relative priorities. Based on thissystem model, we specify the MORL problem in the formof a general MOMDP (Section V-A).
• We propose a novel MORL algorithm, Runtime MORL,
that reuses pre-trained SORL policies for dynamic tradeoffs at runtime (Section V-B).
• We propose another MORL algorithm, Off-Policy MORL,
that learns energy-neutral policies tabula rasa (within reasonable computing costs) and is also capable of dynamic
runtime tradeoffs (Section V-C).
We give a brief review of related literature in Section II.
We explain our experimental setup in Section VI and evaluate
our proposed framework extensively through simulations of
single-task and dual-task EHWSNs. We discuss the resultsof our experiments in Section VII before concluding with
Section VIII.
II. R ELATED W ORK
Multi-objective optimization have traditionally focused
mostly on network performance rather than node energyneutrality [10]. Node-centric RL solutions have primarily
focused on using scalarized reward functions [8], [11], [12].
Other sophisticated MORL algorithms that overcome the limitations of scalarization and enable a posteriori tradeoffs exist
but at extremely high learning and computation costs [13],[14]. Our proposed method cuts down these costs significantly(by at least an order of magnitude). To our knowledge, ourwork is the first to use “true” MORL for ENO in EHWSNs andenable tradeoffs. Consequently, we have yet to address issuesof intermittent computing, task dependency and network-based
optimizations. More importantly, there is still work required to
implement our MORL solution in very resource-constrained
IoT systems. Some possible directions for this would be touse tabular methods [5], [6], linear function approximation[7], [8] or distributed learning [15]; since our framework istheoretically applicable for tabular RL or other less computeintensive RL. It may also be possible to compress the neuralnetworks to fit within given SoC requirements. There havebeen encouraging results where researchers have been able
to compress and fit a 16 GB ImageNet model in a microcontroller (less than 1 MB) [16] within reasonable limits ofaccuracy and latency. In [9], [17], the authors implement RL in
low-power hardware. With more efficient learning algorithms

Energy Flow

Signal Flow

Harvesterℎ𝑡Battery𝑏𝑡

Forecast
𝑓𝑡

MORL
Agent

Priority
𝜔𝑡𝑖

Conformity
𝑘𝑡𝑖

Tasks

Demand𝑑𝑡𝑖

Node Tasks
Sense

Comm. Processing

MORL Framework
ENP Reward

𝑢𝑡𝐸𝑁𝑃 = 𝑈 𝐸𝑁𝑃 (𝑒𝑡 )Task Reward

𝑢𝑡𝑖 = min

1, 𝑈 𝑖𝑧𝑡𝑖

𝑑𝑡𝑖

Energy Budget
𝑧𝑡𝑖

Fig. 1: A general system model for multi-task EHWSNs.
and power-efficient and powerful processors, it may soon bepossible to implement our MORL in small IoT edge devices.
III. S YSTEM M ODELOur system model (Figure 1) assumes a harvest-store-use
solar EHWSN that is required to be always on. The EHWSN
is equipped with an energy harvester and a finite energy
buffer. The node receives user requests to execute different
tasks related to sensing, communication and processing. Eachrequest has a corresponding energy demand required to meetdifferent performance requirements such as sensing rates [4]and throughputs [5], [7].
The MORL agent works in discrete timesteps. It intelligently allocates the energy among different tasks to generate task-utilities and ultimately maximize the node-utility.
It observes its environment (the system state) at timestep t
to decide the conformity of ith task kti ∈ [ 0, 1] , to itsiidemand dit ∈ [ zmin
, zmax] . The energy allocated to taski
i is zti = max(zmin, dit × kti ) which generates a task-utilityof uit = min(1, U i (zti /dit )), where U i is a monotonicallynon-decreasing function. The constraint uit ≤ 1 accounts forthe fact that over-provisioning does not lead to increased
utility. Traditional solutions usually overlooked the issue oftask utilities and opportunistically maximized duty cycles [6],[15]. Our system model is more general and realistic because
it maximizes energy consumption only as requested.
A. Energy Neutral Operation
At time t, the battery level is bt ∈ [ 0, bmax ] , theharvestedis ht ∈ [ 0, hmax ], the total energy demand
Penergy
n
i
is
d
=
d
and
the energy consumed by the node is zt =
t
t
i=1
Pn
i
i=1 zt s.t. zmin ≤ zt ≤ zmax . The energy dynamics of thesystem is therefore given by bt+1 = min(bt +(ht −zt ), bmax ).
Perfect energy-neutrality is guaranteed if ht − zt = 0∀t.
However, this may not always be possible or desirable. The
sensor may need to operate during the night (ht < zt ); or, it
may not be optimal to greedily use all harvested energy (ht =zt ) because some of that energy could be saved to extract moreutility in the future. However, when E[z] = E[h], E[·] being

the expectation operator, the node can be considered energyneutral. The node is operational at time t if bt + ht ≥ zmin .
Otherwise, there is a downtime. In such a case, the node is nonoperational and uses all of the harvested energy to replenish
its battery to a user-specified level and then resumes operation.
Ideally we would like an energy management scheme thatminimizes downtimes but ensures that all the energy harvested
is intelligently used to maximize utility. The energy-neutralityof the node at time t is given
Pt by the Energy Neutral Performance (ENP) metric et = 0 B(ht − zt ) = b0 − bt , expressedas the total difference between harvested and consumed energy[5], [6], [15].
P
The ENP-utility is uEN, given by a function U EN P (et )twhich reflects the energy-neutrality of the node. The nodeutility is a weighted sum of the individual task-utilities w.r.t.
their priorities. For an EHWSN with n-tasks, the relative
priorities between its n + 1 objectives is expressed by ωt =is the priority/preference(ωt1 , ωt2 , ..., ωtn+1 ) where ωti ∈ [0, 1] Pnfor the ith objective. ωtn+1 = (1 − i=1 ωti ) is the impliedpriority for the energy-neutrality of the node. With a slightabuse of notation, for single-task EHWSNs, the priorities oftask maximization and ENO are represented by ωt and (1−ωt ).
The agent’s ultimate
Pn+1 objective is to maximize the overall nodeutility wt = i=1 uit ωti .
IV. T HEORETICAL BACKGROUND

A. Single-objective RL
In standard SORL, at each timestep, an agent observes its
environment state st ∈ S and executes an action at ∈ A
according to some policy π(a|s). Consequently, it receives ascalar reward rt that reflects the optimality of the action w.r.t.
the optimization objective and transitions to the next state s′ .
The rewards are given by the reward function R(s, a, s′ ) andare discounted by a factor of γ ∈ [0, 1]. This process iterates
until the agent reaches a terminal state (end of an episode) andthe process restarts. Using the rewards and its past experiences,
the agent learns increasingly better policies that maximize its
cumulative reward or return.
The Q-value of a state-action pair Qπ (s, a), w.r.t. policy π,
gives the expected return when executing action a from state s
as defined in Equation (1a) for an episode of length T . It canbe can be computed recursively by using the Bellman equationin Equation (1b).
#" TX
γ t R(st , at , st+1 )|s0 = s, a0 = a (1a)Qπ (s, a) = Eπt=0Qπ (s, a) = Eπ [R(s, a, s′ ) + γQ(s′ , π(s′ ))](1b)Thus, RL basically involves i) learning the Q-values of allstate-actions pairs (predictive knowledge) and ii) learning a
policy to choose an action that maximizes the Q-value forevery state (procedural knowledge). We approximate the Qvalues with a neural function approximator Qθ (s, a) parameterized by θ (the critic). The policy is output by a function
πφ (s) with parameters φ (the actor). In this work we use

Fig. 2: The Pareto-front changes when the agent transitions
form state s1 to s2 . Crosses indicate Pareto-dominated actions.
The marker size represents the utility of the action.
Deep Deterministic Policy Gradient (DDPG) [18], to learn
the actor and critic functions. We choose DDPG because it
accommodates continuous states and actions and is simple toimplement (compared to other similar RL algorithms).
B. Multi-objective RL
In a general multi-objective problem, an optimizing variable
is mapped by objective functions into a multi-dimensional
objective space. Pareto-optimal points correspond to thosemappings where the value of one objective cannot be increased
without a decrease in the value of at least one another objective. The overall utility of the variable is a linear combinationof its mapped objective values according to the user-defined
weights (non-linear utility functions are outside the scope of
this work).
In our MORL framework, the optimizing variable is theagent’s action π(s) = a. The Q-functions Qπ (s, a) (and not
the instantaneous reward) are the objective functions because
they represent the long-term effect of the actions. However,Qπ (s, a) is dependent on the state s, which changes at every
timestep. This is shown in Figure 2 where the same set ofsampled actions map to different points in the objective spacewhen the state changes from s1 (blue) to s2 (orange). It isdue to this non-static objective space that traditional elitist
multi-objective optimization methods such as Monte Carlo(MC) rollouts or Evolutionary Algorithms (EA) are infeasible.
These methods require multiple rollouts and generations which
consume a lot of time and resources.
In general, MORL solutions need to learn the Q-functions
and a policy that can infer the optimal action from Paretofrontier. A general Multi-Objective Markov Decision Process (MOMDP) for MORL with m objectives is defined by(S, A, P, R, γ, Ω). S is the continuous state space, A is thecontinuous action space and P(s′ |s, a) defines the transition
probability from state s to s′ as a result of action a. We choose
continuous states and actions for generality (as compared to
MDPs with discrete state-actions spaces in previous works [5],[6]). The reward function is now a vector function, R(s, a) =[R1 (s, a), R2 (s, a), ..., Rm (s, a)] and the discount factors foreach objective are given by γ = [γ1 , γ2 , ..., γm ]. Ω determines
the space of preferences among the m objectives. The MORL
policy µ is now associated with an m-dimensional vector Qvalue function Qµ = [Q1 , Q2 , ..., Qm ] whose components
correspond to the different objective functions. µ extracts thePareto-optimal action w.r.t. the user-preference ω ∈ Ω. In
the following section, we develop a suitable MOMDP for
EHWSNs and present solutions to learn the vector Q-functions
and extract the optimal actions.

Action Space, (𝑛 = 1)
greedy task
actionaction, 𝑘

V. P ROPOSED MORL F RAMEWORK
A. MOMDP Formulation
Multiple Reward Functions: Modern EHWSNs requires a
reward function that accommodates multiple objectives. Previous SORL methods project these objectives into a scalarthat introduces noise and obfuscates the reward incentives.
The most significant drawback is that tradeoffs cannot bemade at runtime because it requires a priori knowledge of
the weights [11]. Another disadvantage is that scalarization
involves complicated reward shaping [5], [12] and introduces
significant design bias.
To avoid these limitations, our framework learns frommultiple separate independent reward signals. Each of thesereward signals represent one and only one optimization objective without any complicated reward shaping. We define therewards for each task by their task-utilities and for energyneutrality by the ENP-utility.
Robust State-Action Space: The state at time t is defined as atuple st = (τ, bt , b¯t , ht , ft , dt ). ht is the harvested energy anddt = (d1t , d2t , ..., dnt ) is a tuple of the different task requests.
We also include additional temporal information,
τ whichPT
represents the time of the day, b¯t =
b
/T
which
k=0 t−kis a moving average of battery values over a horizon T , andft ∈ [0, 1] which is a rough prediction of future energy supply
similar to [6].This addition of temporal information makes
learning process more robust and stable than previous works[5], [8] which didn’t include sufficient temporal informationin their state definitions.
We define the action as the conformity of the node todifferent task demands in our formulation. For an n-task
EHWSN, the action is kt = (kt1 , kt2 , ..., ktn ) where kti isthe conformity of the node to the request dit . This definition
ensures actions are safe in that it never over-provisions energy
and thus prevents unwanted battery depletion. It minimizes theeffects of catastrophic policies even if the learning fails unlike
actions definitions in previous methods [5], [6], [8].
B. Runtime MORL
We now describe our first algorithm, Runtime MORL (Algorithm 1). An n-task EHWSN has an n-dimensional actions
space i.e., |k| = n. It has to optimize over n + 1 objectives where the n + 1th objective is w.r.t. energy-neutrality.
Algorithm 1 uses n pre-trained greedy actor-critic networks(πi , Qi ) to output each component of the action space. πi
outputs the greedy conformity gi for the ith task. An additional
pre-trained greedy actor-critic (πEN P , QEN P ) outputs thetotal P
conformity of the node gEN P w.r.t the total demandnd = i=1 di . We can
Pnconstruct a plane in the action space
given by gEN P =i=1 ki that represents all actions thatare greedily energy-neutral. We then determine the convex
hull that contains this plane and all the greedy actions. The

𝑔1

sampledactions𝑥1

greedy ENP
action𝑔𝐸𝑁𝑃

Objective Space
Task Objective,
𝑄1

𝑥1 has the highest
utility for a given 𝜔

𝑥2

Pareto-front
ENP Objective, 𝑄𝐸𝑁𝑃

Fig. 3: As we move from one greedy action to another in
the action space, we traverse along the Pareto-optimal frontierin the objective space. Runtime MORL samples actions (redcrosses) from in between the greedy actions, g1 and gEN P .
main intuition behind Algorithm 1 is that as we move fromone greedy action to another along this convex hull, we are
actually traversing along the Pareto-front in the objective space
making tradeoffs. This is valid in our case because taskutility functions are monotonically non-decreasing function ofconformity (or action).
1) Minimizing Computation Costs: To find the optimal action k ∗ we sample the locally approximate Paretooptimal from the convex hull. For a given user-preferenceω = (ω1 , ω2 , ..., ωEN P ), the algorithm samples j actions
x1 , x2 , ..., xj . The Q-values for all actions are evaluated bylinear combination of outputs of the critics Qi and QEN P ,weighted by the task preferences ωi and ωEN P . The locally
optimal action is the one with the maximum value. Thisway, a change in ω at runtime only requires inferring thecritics and a recomputation of the action-values, which is
not a very costly operation. Overall, this algorithm requires
much less computation than having to infer the entire Paretooptimal front and perform gradient descent to find the optimal
action. Another practical advantage of this sampling approachis that we can always find an action at least as good as thegreedy actions without any assumption about the concavity of
the Pareto-frontier. This is significant because some MORL
methods do not perform well for concave Pareto-frontiers.
Figure 3 illustrates Algorithm 1 for a single-task scenario(n = 1). Action g1 greedily maximizes the task-utility byincreasing conformity k, whereas gEN P greedily maintains
energy-neutrality by decreasing k. Here, the convex hull issimply a straight line bounded by g1 and gEN P . Algorithm 1
samples two points on the line indicated by the red crosses.
The Q-values for all actions (greedy and sampled) are mapped
to the objective space using the outputs of the critics (Q1 ,QEN P ) and ω to find the optimal action. The action values
are represented by the size of the circles in the figure.
C. MORL with Off-Policy Corrections
We now consider the case when pre-trained greedy actors
and critics are not available and have to be trained fromscratch. With our second algorithm, Off-policy MORL (Algorithm 2), the actors learn greedy policies, and critics learn

Algorithm 1: Runtime MORL
Input : ω ∈ Ω, priorities for n + 1 objectivess ∈ S, State(Qi , πi ), Greedy actor-critics, i = 1, 2, .., n(QEN P , πEN P ), Energy-neutral actor-critic
Output: Action k ∗ that maximizes utility, |k ∗ | = n
1 actionList = [ ]// Proposed actions list
2 for i = 1, 2, ...n, EN P do
3
gi = πi (s)// Greedy Actions
4
actionList.append(gi )5 end for6 Sample j actions x1 , x2 , ...xj from the convex hull ofactions in actionList and append to actionList
7 W ={ }8 for x in actionList do
Pn+1
9
W [x] = i=1 Qi (s, x)ω i
10 end for∗
11 return k = argmax W [x]Algorithm 2: Off-policy MORL
Input : γi , discount factor for ith objective,i = 1, 2, ...n + 1
Initialize: Randomly initialize actor-critic pairs(Qi , πi ) and empty replay buffer D
1 for episode = 1, L do
2for t = 1, T do// T = episode length3
Observe the preferences ωt ∈ Ω
4Observe state st
5
Select action at using Algorithm 1
6
Execute action at , observe the reward vector
rt = [rt1 , rt2 , ..., rtm ] and the next state st+1
7
Store (st , at , r t , st+1 , ωt ) in D
8
Sample minibatch B of N transitions from D
9for i = 1, n + 1 do
10
Update Qi using ri and γi in Equation (2),11
Update πi
12
end for13
end for
14 end forxtheir Q-values from the experience samples generated by theagent’s interaction with the environment. The agent executes
actions that lie in between the greedy actions depending on thepreference ω using the method in Algorithm 1. This means thatthe critics have to learn the Q-values Qi for a greedy policy
πi from transitions generated by a different policy, say µ.
To account for this, we note that the probability of statetransitions from s to s′ are different for πi and µ. Thus, weapply an off-policy correction by multiplying the expected
Q-values of the next state ′by their ratio of their transition|s,πi )
probabilities, ρi (s, s′ ) = P(sP(s′ |s,µ) . For our case, we approximate ρi (s, s′ ) ≈ ωi . Although theoretical guarantees for thisare beyond the scope of this work, this approximation makes
sense. When ωi is closer to unity, it implies that µ is more
dominated by πi so the off-policy correction does not change
the expectation much. On the other hand, when ωi is very low,
µ is much further away from πi and thus the expected returnis reduced proportionately.
′i′Qi (s, a) = Eπi [r + γi Qi (s , πi (s ))]= Eµ [ri ] + γi ρi (s, s′ )Eµ [Qi (s′ , πi (s′ ))]
i

i

′

(2)

′

≈ Eµ [r + γi ω Qi (s , πi (s ))]VI. E VALUATION S ETUP
A. Simulation Environment
We evaluate our framework by simulating solar EHWSN
systems based on the solar radiation data from 1995 to 2018,logged hourly by outdoor rooftop pyranometers [19]. We
therefore set the time interval for one timestep to be one hour.
We use the first ten years (1995-2004) for training the RL
agents and the remaining years to test their performance. We
use an episodic MDP - with one episode lasting one day (or 24
timesteps). An episode abruptly terminates with zero reward

TABLE I: Simulation Parameters
Parameter
Battery, bt
Harvester, ht
EHWSN, zt
Requests, dt

maximum value
bmax
hmax = 5% of bmax
zmax = 5% of bmax
zmax = 5% of bmax

minimum value
bmin = 10% of bmax
hmin = 0zmin = 0.5% of bmax
zmin = 0.5% of bmax

when a downtime occurs. The parameters of the evaluated
EHWSN system, normalized to bmax , are shown in Table I.
These parameters roughly correspond to a realistic EHWSN[20] with a current rating of 100 mA and a 2000 mA h battery.
The node goes into recovery mode when the battery capacity
drops below 10%. The requests are randomly generated by
the request function such that E[ht ] ≈ E[dt ]. This ensures that
ENO is actually feasible. A rough estimate of the weather
prediction ft , is given by adding some random noise to therolling average of ht over the next ten days.
B. Utilities and Reward Functions
We run our evaluations on single-task (bi-objective) anddual-task (tri-objective) EHWSNs. The single-task node maximizes its sensing utility by increasing the conformity to theuser demand while maintaining long-term energy-neutrality.
Typically, the utility of sensed data increases linearly with energy consumption [2]–[4], [6]. Thus, we define the rewards forsensing rate maximization by the sense-utility usense, whicht
grows linearly with conformity (energy usage) according thelinear-utility function in Figure 4 (left).
P
The ENP-utility uEN
corresponds to the rewards thattrepresent the energy-neutrality of the node given by(1,
if b¯t ≥ bth
EN P
(3)
ut
= b¯t −bminbth −bmin , otherwise
This reward function is illustrated in Figure 4 (right). b¯t is thethe moving average battery level over ten days and bth is a

TABLE II: Multi-objective ENO agents
MethodOurs
Ours
Scalar Product [5], [8]Scalar Sum [11]Namemorl runtime(2-task)morl multi(3-task)
mul scalar
add scalar

Rewards/Values
ωQπsense + (1 − ω)Qπenp
ωsense Qπsense +ωtx Qπtx +ωEN P Qπenp
usense × uEN P
ωusense + (1 − ω)uEN P

Fig. 4: Linear and concave reward functions quantify theutilities for different tasks such as sensing, communication andprocessing. The ENP-utility function indicates the long-term
energy-neutrality of the node.
user-defined battery threshold. This is similar to the reward
functions in [5], [6], [12], [15] except that it has minimal
reward shaping and uses b¯t instead of bt . bth is fixed at 80%
of bmax for our case but this may be adjusted as necessary.
We use the average battery level here because it reflects thelong-term temporal nature of ENO. For single-task EHWSN, ω
reflects the relative preference between maximizing the senseutility and ENP-utility. ω = 1 maximizes sense-utility andω = 0 emphasizes energy-neutrality.
For dual-task EHWSNs, we consider an EHWSN systemthat needs to sense and transmit its data. In addition toremaining energy-neutral and maximizing its sense-utility, italso has to maximize its throughput. Here, the conformity
k = z/d represents the SNR and the node’s throughput P
is given by Shannon’s capacity formula P = log(1 + k).
The task-utility and reward is therefore utx= P . Highertthroughput requires higher SNR but has diminishing returns.
This relationship is represented by the concave-utility reward
function in Figure 4. For sake of example, this reward function
assumes Binary Phase Shift Keying (BPSK) modulation withadditive white Gaussian noise and Rayleigh fading. At every
instant, the dual-task node has to decide how much energy
it must allocate to transmission and sensing tasks so as tomaximize its throughput (or tx-utility) utx
t and sense-utility
P
).
,
while
ensuring
long-term
ENO(maximizinguEN
usensett
The preference between the objectives is given by a tupleω = (ωsense , ωtx , ωEN P ) s.t. ωEN P = 1 − (ωsense + ωtx ).
C. Metrics
We compare the task-utilities between different solutions
by their annual average. The energy-neutrality of different
methods is compared on the basis of the total number oftimes the node goes into recovery mode (downtimes). Thisis a more direct representation of the actual energy-neutrality
than the ENP-utility and facilitates fair comparisons. An
intelligent policy finds the right balance between maximizingtask-utilities and minimizing downtimes. Since the training
period is the same for all RL agents, the learning costs are
given by the number of downtimes that happen during thetraining period, lesser being better. Ideally, we would like thenode to learn an optimal energy management policy with as
few downtimes as possible.

Fig. 5: Agent enp (ours) has similar energy-neutrality (downtimes) as the most energy-neutral policy max enp. sense (ours)
sacrifices some of its energy-neutrality to increase its utility.
VII. E XPERIMENTAL R ESULTS
Each experiment is executed ten times with different seeds.
The average over all seeds are used for comparison. Theinterquartile range (IQR) is indicated by error bars (not shownin some figures for clarity). A summary of the different MORL
agents we evaluate is given in Table II.
A. Baselines
In Figure 5, we compare between different RL policies
against a baseline heuristic policy called max enp (blue) w.r.t.
their sense-utilities (top) and downtimes (bottom). max enp
is the most energy-neutral policy with the minimum possible
downtimes. max enp increases node conformity k monotonically with rising battery levels in a non-linear fashion. The
hyperparameters for this function were empirically determinedusing greedy search and non-causal information. No policy
can extract higher utility than max enp without increasing thenumber of downtimes.
Figure 5 also shows three SORL agents (γ = 0.997):
sense (red), enp (green) and mul scalar (purple). sense andenp maximize sense-utility and ENP-utility respectively using
the linear-utility and ENP-utility reward functions (Figure 4).
×mul scalar uses a scalarized reward given by rt = usensetPuEN
similar to [8], [12].
t
We observe that sense has the highest utility but also
correspondingly highest number of downtimes. This is due
to its reward function that is designed solely to maximize its
utility. Although sense extracts higher utility within acceptable
limits of energy-neutrality, we cannot say whether it is optimal.
A more optimal solution would achieve a higher sense-utility

Fig. 6: morl runtime(ω) (Algorithm 1) can tradeoff senseutility with energy-neutrality (downtimes) at runtime.
while maintaining the same number of downtimes as sense.
It is difficult to define absolute optimal baselines in this caseand so we use sense as the baseline for comparing sense-utility
among RL agents.
On the other hand, enp has the lowest utility and lowest
number of downtimes on account of its ENO-centric reward
function. Although enp has lower utility than max enp, theenergy-neutrality (downtimes) of enp and max enp are similar(green and blue bars). This means that enp is performing nearoptimally w.r.t. the ENO objective. These observations lead us
to conclude that the SORL agents are learning policies thatare consistent with their reward functions.
The performance of mul scalar agent lies somewhere in
between that of sense and enp. Due to its scalarized reward
function, mul scalar is neither as energy-neutral as enp nor as
useful as sense. This is a limitation of scalarization methods
for SORL. It shows that while one can end up with aworking policy with scalarization, due to the ”mixing” ofrewards, it is not fully clear what objective is actually being
optimized. Furthermore, it is not at all possible to tradeoff
between different objectives with the reward scheme used in
mul scalar.
B. Scalarization and Runtime Tradeoffs
We now consider the dual-objective case when the singletask EHWSN agent has to tradeoff between maximizing thePsense-utility usense, and the ENP-utility uEN. For a giventtω, the agent has to maximize the node-utility, wt = ωusense+tP(1 − ω)uEN.
t
We compare between two RL agents: morl runtime(ω)
and add scalar(ω). morl runtime(ω) uses reuses the sense
and enp agents from the previous section and our proposed
Runtime MORL algorithm (Algorithm 1) to generate tradeoff
policies. add scalar(ω) uses scalarized reward functions givenPby rt = ωusense+ (1 − ω)uEN, similar to [5], [11], [12].
tt
The important distinction between these two methods is thatmorl runtime(ω) can tradeoff between objectives even if ω isprovided only at runtime. In contrast, add scalar(ω) needs ω
to be known before the training process and has to be retrained

Fig. 7: morl multi agent has lower downtimes when ENO has
higher priority for ENO (green and red) than when it is low(blue and orange). Similarly, for the same ωEN P , the agent
trades off between sense-utility and tx-utility. (2015-2019 notshown for clarity.)
every time ω changes. Scalarization methods are very limitedin their application precisely because of this requirement ofprior knowledge of ω.
Figure 6 shows the test results of morl runtime(ω) andadd scalar(ω) for different values of ω (held constant during
testing). We observe that our morl runtime(ω) agent canindeed tradeoff for different ω. High priority (ω = 0.8,
brown) increases utility and downtimes and vice versa for low
priority (ω = 0.2, blue). Similar tradeoffs are achieved by
add scalar(ω). Since we are plotting the annual average overten different seeds, the differences in the utility do not seemvery significant. However, when we consider the cumulative
utilities over long periods of operation, the differences arequite substantial.
We note that the effect of tradeoffs is lesser formorl runtime than add scalar. This is because the tradeoff
takes place in the Q-value space for morl runtime and in
the reward space for add scalar. This is actually preferable
because morl runtime is more optimal than add scalar. In
Figure 6, morl runtime (0.2) (blue) consistently extracts muchhigher utility than add scalar (0.2) (cyan) for similar downtimes. In contrast, add scalar (0.8) (orange) has disproportionately higher downtimes for only a slight increase in senseutility than our morl runtime (0.8) (brown).
C. MORL for three objectives
Finally, we consider the case of the dual-task EHWSN.
We use Algorithm 2 to train an agent tabula rasa to optimize and tradeoff between its three objectives (sensing,transmission and ENO). During training, the user priorityω = (ωsense , ωtx , ωEN P ) changes randomly at each timestep.
At test, we maintain a constant ω for each experiment forcomparison purposes.

R EFERENCES

Fig. 8: morl multi has acceptable learning costs (slightly
higher than sense) even though the learning problem is muchharder.
First, we observe how morl multi adapts its energyneutrality with changing ω (Figure 7. When energy-neutrality
has low priority (ω = (ωsense , ωtx , 0.1)), the downtimes (bottom figure) are much higher than when ω = (ωsense , ωtx , 0.7)
(blue and orange vs green and red). Secondly, we observe
how the agent trades off between sense-utility and tx-utility
when ωEN P = 0.1. In the top and middle figures, the bluelines corresponds to a higher sensing priority than transmission(ωsense > ωtx ). As a result, the blue line scores higher on
the sense-utility (top) and lower on the tx-utility (middle)
than the orange line (ωsense < ωtx ). This clearly shows thetradeoff behavior. A similar observation can be made for whenωEN P = 0.7 (green and red lines).
We now turn our attention to the increased learning costsof our MORL solutions. Figure 8 shows the learning costs
for sense and enp, which is competitive with previous SORL
methods [15]. The other agents, mul scalar and add scalar,also have similar learning costs (not shown). What is interesting is that morl multi also has similar learning costs in spiteof having to learn three different greedy actor-critics within
the same training period as previous SORL methods. This is
primarily due to off-policy corrections in Algorithm 2 andauto-regulation among the greedy agents in Algorithm 1.Thus,
our framework can learn to optimize between three objectives
successfully in dual-task EHWSNs, without a drastic increasein learning costs.
VIII. C ONCLUSION
Modern EHWSNs are complex SoCs that need to optimize
multiple objectives to maximize their utility and maintain
ENO. Traditional approaches cannot handle multiple objectives and tradeoff between them at runtime. Our proposed
MORL framework can learn policies and perform tradeoffs
within feasible learning costs. Simulations on single-task
and dual-task EHWSNs show that our framework results innear-optimal policies that can dynamically tradeoff between
objectives at runtime. By compromising between computeintensive MORL methods and elitist MC/EA methods, our proposed MORL algorithms are a feasible solution for resourceconstrained EHWSN.
ACKNOWLEDGMENT
This work was partially supported by JST CREST Grant
Number JPMJCR20F2 and JSPS KAKENHI Grant Number
18J20946.

[1] D. Ma, G. Lan, M. Hassan, W. Hu, and S. K. Das, “Sensing, computing, and communications for energy harvesting iots: A survey,” IEEE
Communications Surveys & Tutorials, vol. 22, no. 2, pp. 1222–1250,
2019.
[2] A. Kansal, J. Hsu, S. Zahedi, and M. B. Srivastava, “Power management
in energy harvesting sensor networks,” ACM Transactions on Embedded
Computing Systems (TECS), vol. 6, no. 4, p. 32, 2007.
[3] C. M. Vigorito, D. Ganesan, and A. G. Barto, “Adaptive control ofduty cycling in energy-harvesting wireless sensor networks,” in 2007
4th Annual IEEE Communications Society Conference on Sensor, Meshand Ad Hoc Communications and Networks. IEEE, 2007, pp. 21–30.
[4] K. Geissdoerfer, R. Jurdak, B. Kusy, and M. Zimmerling, “Getting
more out of energy-harvesting systems: energy management under timevarying utility with preact,” in Proceedings of the 18th International
Conference on Information Processing in Sensor Networks. IEEE, 2019,
pp. 109–120.
[5] R. C. Hsu, C.-T. Liu, and H.-L. Wang, “A reinforcement learningbased tod provisioning dynamic power management for sustainable
operation of energy harvesting wireless sensor node,” IEEE Transactions
on Emerging Topics in Computing, vol. 2, no. 2, pp. 181–191, 2014.
[6] S. Shresthamali, M. Kondo, and H. Nakamura, “Adaptive power management in solar energy harvesting sensor node using reinforcement
learning,” ACM Transactions on Embedded Computing Systems (TECS),
vol. 16, no. 5s, p. 181, 2017.
[7] A. Ortiz, H. Al-Shatri, X. Li, T. Weber, and A. Klein, “Reinforcement
learning for energy harvesting point-to-point communications,” in 2016
IEEE International Conference on Communications (ICC). IEEE, 2016,
pp. 1–6.
[8] F. A. Aoudia, M. Gautier, and O. Berder, “Rlman: an energy manager
based on reinforcement learning for energy harvesting wireless sensor
networks,” IEEE Transactions on Green Communications and Networking, vol. 2, no. 2, pp. 408–417, 2018.
[9] F. Fraternali, B. Balaji, Y. Agarwal, and R. K. Gupta, “Aces–automaticconfiguration of energy harvesting sensors with reinforcement learning,”
arXiv preprint arXiv:1909.01968, 2019.
[10] Z. Fei, B. Li, S. Yang, C. Xing, H. Chen, and L. Hanzo, “A surveyof multi-objective optimization in wireless sensor networks: Metrics,
algorithms, and open problems,” IEEE Communications Surveys &Tutorials, vol. 19, no. 1, pp. 550–586, 2016.
[11] P. V. R. Ferreira, R. Paffenroth, A. M. Wyglinski, T. M. Hackett,
S. G. Bilén, R. C. Reinhart, and D. J. Mortensen, “Multiobjective
reinforcement learning for cognitive satellite communications using
deep neural network ensembles,” IEEE Journal on Selected Areas inCommunications, vol. 36, no. 5, pp. 1030–1041, 2018.
[12] Y. Rioual, Y. Le Moullec, J. Laurent, M. I. Khan, and J.-P. Diguet,“Reward function evaluation in a reinforcement learning approach forenergy management,” in 2018 16th Biennial Baltic Electronics Conference (BEC). IEEE, 2018, pp. 1–4.
[13] R. Yang, X. Sun, and K. Narasimhan, “A generalized algorithm for
multi-objective reinforcement learning and policy adaptation,” in Advances in Neural Information Processing Systems, 2019, pp. 14 636–
14 647.
[14] M. Pirotta, S. Parisi, and M. Restelli, “Multi-objective reinforcement
learning with continuous pareto frontier approximation,” in Twenty-NinthAAAI Conference on Artificial Intelligence, 2015.
[15] S. Shresthamali, M. Kondo, and H. Nakamura, “Power managementof wireless sensor nodes with coordinated distributed reinforcement
learning,” in 2019 IEEE 37th International Conference on Computer
Design (ICCD). IEEE, 2019, pp. 638–647.
[16] J. Lin, W.-M. Chen, Y. Lin, J. Cohn, C. Gan, and S. Han, “Mcunet: Tiny
deep learning on iot devices,” arXiv preprint arXiv:2007.10319, 2020.
[17] F. Restuccia and T. Melodia, “Deepwierl: Bringing deep reinforcement
learning to the internet of self-adaptive things,” pp. 844–853, 2020.
[18] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa,
D. Silver, and D. Wierstra, “Continuous control with deep reinforcement
learning,” arXiv preprint arXiv:1509.02971, 2015.
[19] Japan Meteorological Agency, “Japan meteorological agency,” 2019,[Online; accessed 6-July-2019]. [Online]. Available: http://www.jma.
go.jp/jma/index.html[20] “Waspmote: The sensor platform to develop iot projects — libelium,” https://www.libelium.com/iot-products/waspmote/, (Accessedon 01/22/2021).