Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints
Authors: David Henry Mguni
Aivar Sootla
Juliusz Krzysztof Ziomek
Oliver Slumbers
Zipeng Dai
Kun Shao
Jun Wang
Venue: NeurIPS 2022 Submitted (2022)

Abstract: Many real-world settings involve costs for performing actions; transaction costs in financial systems and fuel costs being common examples. In these settings, performing actions at each time step quickly accumulates costs leading to vastly suboptimal outcomes. Additionally, repeatedly acting produces wear and tear and ultimately, damage.  Determining when to act is crucial for achieving successful outcomes and yet, the challenge of efficiently \textit{learning} to behave optimally when actions incur minimally bounded costs remains unresolved. In this paper, we introduce a  reinforcement learning (RL) framework named Learnable Impulse Control Reinforcement Algorithm (LICRA), for learning to optimally select both when to act and which actions to take when actions incur costs. At the core of LICRA is a nested structure that combines RL and a form of policy known as \textit{impulse control} which learns to maximise objectives when actions incur costs. We prove that LICRA, which seamlessly adopts any RL method, converges to policies that optimally select when to perform actions and their optimal magnitudes. We then augment LICRA to handle problems in which the agent can perform at most $k<\infty$ actions and more generally, faces a budget constraint. We show LICRA learns the optimal value function and ensures budget constraints are satisfied almost surely. We demonstrate empirically LICRA's superior performance against benchmark RL methods in OpenAI gym's Lunar Lander and in Highway environments.

Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints
Anonymous Author(s)
Abstract
Many real-world settings involve costs for performing actions; transaction costs in financial systems and fuel costs being common examples. In these settings, performing actions at each time step quickly accumulates costs leading to vastly suboptimal outcomes. Additionally, repeatedly acting produces wear and tear and ultimately, damage. Determining when outcomes and yet, the challenge of efficiently learning to behave optimally when actions incur minimally bounded costs remains unresolved. In this paper, we intro- duce reinforcement learning (RL) framework named Learnable Impulse Control Reinforcement Algorithm (LICRA), for learning to optimally select both when to act and which actions to take when actions incur costs. At the core of LICRA is a nested structure that combines RL and a form of policy known as impulse control which learns to maximise objectives when actions incur costs. We prove that LICRA, which seamlessly adopts any RL method, converges to policies that optimally select when to perform actions and their optimal magnitudes. We then augment LICRA to handle problems which the agent can perform at most oo actions and more generally, faces a budget constraint. We show LICRA learns the optimal value function and ensures budget constraints are satisfied almost surely. We demonstrate empirically LICRA's superior performance against benchmark RL methods in OpenAI gym's 's Lunar Lander and in Highway environments and a variant of the Merton portfolio problem within finance.
2
crucial for achieving successful
10
11
13
14
15
17
19
Introduction
21
There many settings in which agents incur costs each time they perform an action. Transaction 23 costs in financial settings [19], fuel expenditure [32], toxicity as a side effect of controlling bacte- and tear are just 25 few among many examples [13]. In these settings, performing actions at each time step is vastly suboptimal since acting in this way results in prohibitively high costs and undermines the service life of machinery. Minimising wear and tear is an essential attribute to safeguard against failures that can result catastrophic losses [13].
ria [29] and physical damage produced by repeated action that produces
27
28
29 Reinforcement learning (RL) a framework that enables autonomous agents to learn complex behaviours from interactions with the environment [30, 11]. Within the standard RL paradigm, determining optimal actions involves making a selection from among many (possibly infinite) actions; procedure that must be performed at each time-step the agent decides an action. In unknown settings, the agent cannot immediately exploit any topological structure the action set (if any exists). Consequently, learning not take action i.e performing a
31
33
null action, involves 35 expensive optimisation procedures over the entire action set. Since this must be done at each state, this process vastly inefficient for learning optimal policies when the agent incurs costs for acting.
37 In this paper, we tackle this problem by developing an RL framework for finding both an optimal criterion to determine whether not to execute actions as well as learning optimal actions. A key 39 component of our framework is a novel combination of RL with form policy known as impulse
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

40 control [22, 19]. This enables the agent to determine the appropriate points to perform an action as well as the optimal action itself. Despite its fundamental importance as a tool for tackling decision problems with costly actions, presently, the use of impulse control within learning contexts (and unknown environments) unaddressed.
41
43
44
We present an RL impulse control framework called LICRA, which, to our knowledge, is the first learning framework for impulse control. To enable learning optimal impulse control policies in unknown environments, we devise framework that consists of separate RL components for learning when to act and how to act optimally. The resulting framework is a structured two-part learning process which differs from current RL protocols. In LICRA, at each time step, the agent firstly makes a decision whether to act or not leading to a binary decision space {0. 1] (we later show that this
45
46
47
48
49
50 is determined by evaluating an masy-to-evaluate criterion which has the value function as its input).
51
The second decision part determines the best action to take. This generates subdivision of the state 52 space into two regions; one in which the agent performs actions and another in which it does not act
53 at all. This is extremely useful since the agent quickly determines the set of states to not take actions
54 while performing actions only at the subset of states where actions are to be executed.
55 We then establish theory that ensures convergence of LICRA an optimal policy for such settings.
56 To do this, we give a series of results namely:
i) We establish dynamic programming principle (DPP) for impulse control and show that the optimal value function can be obtained as a limit of a value iterative procedure (Theorem 1) which lays the foundation for RL approach to impulse control.
58
59
60
ii) We extend result i) to a new variant of Q learning which enables the impulse control problem to be
61
solved using RL method (Theorem 2).
62
iii) We characterise the optimal conditions for performing an action which we reveal to be a simple obstacle condition' involving the agent's value function (Prop. 1). Using this, the agent can quickly determine whether or not it should act and if so, then learn what the optimal action is.
63
64
65
iv) We then extend the result i) (linear) function approximators enabling the value function to be parameterised (Theorem 3).
66
87
iv) In Sec. 6, we extend LICRA include budgetary constraints so that each action draws from a fixed budget which the agent must stay within. Analogous to the development of i), we establish another DPP from which we derive a Q-learning variant for tackling impulse control with budgetary constraints (Theorem 4). A particular case of a budget constraint is when the number of actions the
68
69
70
71
agent can take over the horizon capped.
Lastly, we perform set of experiments to validate our theory within the Highway driving simulator and OpenAl's LunarLander [7].
73
74
LICRA confers a series of advantages. As we demonstrate in our experiments, LICRA leams to compute the optimal problems in which the agent faces costs for acting in an efficient way which outperforms leading RL baselines. Second, as demonstrated in Sec. 6 LICRA handles settings in which the agent has a cap the total number of actions is allowed to execute and more generally, generic budgetary constraints. LICRA is able to accommodate any RL base algorithm unlike various RL methods designed to handle budgetary constraints.
75
78
77
78
79
2 Related Work
80
81 In continuous-time optimal control theory [24], problems in which the agent faces a cost for each
82 action are tackled with a form of policy known as impulse control [22, 19, 2]. impulse control frameworks, the dynamics of the system are modified through a sequence of discrete actions or bursts chosen at times that the agent chooses to apply the control policy. This distinguishes impulse control 85 models from classical decision methods in which an agent takes actions at each time step while being
83
84
86 tasked with the decision of only which action to take. Impulse control models represent appropriate modelling frameworks for financial environments with transaction costs, liquidity risks and economic environments in which players face fixed adjustment costs (e.g. memu costs) [16, 20].
88
89 The current setting is intimately related to the optimal stopping problem which widely occurs in
90 finance, economics and computer science [23, 31]. In the optimal stopping problem, the task is to
91 determine criterion that determines when to arrest the system and receive a terminal reward. In this
32 case, standard RL methods are unsuitable since they require an expensive sweep (through the set of states) to determine the optimal point to arrest the system. The current problem can be viewed

---

94
as an augmented problem of optimal stopping since the agent must now determine both a sequence of points to perform an action or intervene and their optimal magnitudes - only acting when the cost of action is justified [25]. Adapting RI. to tackle optimal stopping problems has been widely studied [31, 4, 9] and applied to a variety of real-world settings within finance [12], radiation therapy
95
96
97
98
[1] and network operating systems [3]. Our work serves natural extenstion to RI. approaches to
99
optimal stopping to the case in which the agent must decide at which points to take many actions. As with optimal stopping, standard RL methods cannot efficiently tackle this problem since determining whether to perform a O action requires a costly sweep through the action space at every state [31]. In [66] the authors introduce "sparse action" with similar motivation impulse control. However, the authors treat only the discrete action space case. The authors in [26] do not discuss a broader theoretical framework of dealing with "sparse actions", and develop purely algorithmic solutions. Additionally, unlike the approach taken in [26], the problem setting we consider is one in which the agent faces cost for each action- the produces need for the agent be selective about where it performs actions (but does not necessarily constrain the magnitude or choice of those actions).
101
102
103
105
106
107
3 Preliminaries
108
109
Reinforcement Learning (RL). In RL, an agent sequentially selects actions to maximise its expected
110
returns. The underlying problem is typically formalised as an MDP (S, A, P, R, 7) where S C R' is the set of states, A c R is the set of actions, P: S x A x S + [0. 1] is transition probability function describing the system's 's dynamics, R S A + R is the reward function measuring the
111
112
113
agent's 's performance and the factor E [0, 1) specifies the degree to which the agent's 's rewards are discounted over time [30]. At time & E 0, .., the system is in state 8; E SS and the agent must choose an action E AA which transitions the system to a new state S1+1 ~ P(-|st, at) and produces a reward R(s,, a.). A policy т. S x AA > [0. 1] is probability distribution over state-action pairs where (a]s)) represents the probability of selecting action E A in state E S. The goal of an
114
115
116
117
118
RL agent is to find a policy स E II that maximises its expected returns given by the value function: E| 7² R(8e, ae) lat- T--(st), 80 = s) where II is the agent's 's policy set. The action value function is given by Q(s, a) = -'= R(s,, aa = a, = s).
119
vn (s)
120
We consider setting in which the agent faces at least some minimal cost for each action it performs. With this, the agent's 's task is to maximise:
122
8² (s) = E
rd [R(s;, a:) C(st, at)3 C(s), 30 =
(I)
t=
123
where for any state S and any action a A, the functions R. and C are given by R(s, a) 1011-1.c lae,)) where Lae is the indicator function which is when a E A and 0 otherwise and C(s, a) c(s.a) ((s.a)losA where e S x A - R is a minimally bounded (cost) function
124
R(s, LeA R(s, ()(1
125
128
that introduces a cost each time the agent performs an action. Examples of the cost function is a quasi-linear function of the form c(st, ar) = K + fax) where f A + K- and is a positive real- valued constant. Since acting at each time step would incur prohibitively high costs, the agent must be selective when to perform an action. Therefore, in this setting, the agent's problem augmented to learning both an optimal policy for its actions and, learning at which states to apply its action policy.
127
128
129
130
Example: Merton Portfolio Problem with Transaction Costs [10]. An investor performs series of costly portfolio adjustments by buying and selling amounts of different assets within their portfolio. Each investment incurs fixed minimal cost (also known as transaction costs) which is deducted from the investor's available cash-flow. The investor's s aim is to maximise their total wealth (the value of the sum of their assets) at some time horizon by adjusting their portfolio of investments. Problems of this kind, portfolio investment problems are of fundamental importance within finance [18].
132
133
134
135
136
137
Example An autonomous vehicle must perform a series of actions to perform a task. Each action draws from its fuel budget. In order to complete its task successfully, during the task, the vehicle must ensure maintains an available supply.
138
139
Le. function which is bounded below by a positive constant.
3

---

140
4
The LICRA Framework
141 In RL, the agent's problem involves learning to act at every state including those in which actions do
not significantly impact on its total return. While can add zero action to the action set A and apply standard methods, we argue that this may not be the best solution in many situations. We argue the optimal policy has the following form:
143
(-|s) = {o-0)
at E St, & St,
(2)
which implies that simplify policy learning by determining the set SJ first - the set where actually need to learn the policy.
148
147
We now introduce a learning method for producing impulse controls. This enables the agent to learn to select states to perform actions. Therefore, now agent tasked with learning to act states that are most important for maximising its total return given the presence of the cost for each action. Now at each state the agent first makes a binary decision to decide to perform an action.
148
150
151
Our framework, LICRA consists of two core components: firstly a RL process g SS x {0, + [0, I] and second RL process π S x - [0, 1]. The role of g is determine whether or not an action is to be performed by the policy ι at a given state If activated, the policy π determines the action to be selected. Prior to decisions being made, the policy π communicates to g the action would take. An important feature of our LICRA is the sequential decision process. In LICRA, the policy first proposes an action a E A which observed by the policy g. Therefore, the role of g is to prevent actions for which the change in expected future rewards does not exceed the costs incurred for taking such actions. By isolating a decision policy over whether an action should be taken or not, the impulse controls mechanism results in a framework in which the problem facing the agent has markedly reduced decision space (in comparison to a standard RL method). Crucially, the agent must compute optimal actions at only subset of states which are chosen by the policy g. Below is the pseudocode for LICRA, we provide full details of the code in Sec. of the Appendix.
152
153
154
155
156
157
158
160
162
Algorithm 1: Learnable Impulse Control Reinforcement Algorithm (LICRA)
1:
Input: Stepsize batch size B, episodes K, steps per episode T, mini-epochs
2: Initialise: Policy network (acting) æ, Policy network (switching) g.
Critic network (acting Critii network (switching WW
3: Given reward objective function, R. initialise Rollout Buffers B,, Bg (use Replay Buffer for SAC)
4: for Nepisodez do
5: Reset state Sy, Reset Rollout Buffers B., Bg
6: for = 0, 1,......0 do
7:
Sample 0₂ n (()
8:
Sample 9: ~ g(-
9:
if gt = 0 then
10:
Apply a₂ so 81+1 ~ P(. (a,, st):
11:
Receive rewards ri R( az) Store (st, ar, St+1. rt) ) in B
12:
else
14:
Apply the null action so 8t41 P(-10,st). Receive rewards Ti R(s,, 0).
16:
end if
Store (St. S141, r1) in B,
18:
end for
19:
// Learn the individual policies
20:
Update policy æ and critic V-= networks using Bz
21:
Update policy g and critic Vg networks using Bg
22:
end for
163 While we consider now two policies n, g. the cardinality of the action space does not change. In the discrete case the cardinality is still [A] + where |A] is cardinality of the action space for policy

---

185
Although action space cardinality does not change there are still benefits of using impulse control mechanism. This mechanism forces the agent to first determine the set of states to perform actions only then determine the optimal actions at these states. An important fact to note is that the decision space for the determining whether or not to execute an action S x {0. 1} i.e at each state makes a binary decision. Consequently, the learning process for aspect is much quicker than a policy which
186
167
168
169
170
must optimise over a decision space which is ||||A|| (choosing an action from its action space at every state). This results in the agent rapidly learning which states to focus on to learn which actions to perform. In the case of π with a continuous action space again the impulse control mechanism does not change the cardinality of the action space. However, if the set S/S1, where the optimal policy chooses O, is large enough, then again it can be more efficient to learn g first and only then learn (we later validate this claim empirically, see Sec. 11.2),
172
174
175
176 In Sec. 5, we prove the convergence properties of LICRA. LICRA consists of two independent procedures: a learning process for the policy æ and simultaneously, learning process for the impulse policy g which determines at which states to perform an action. our implementation, we used proximal policy optimisation (PPO) [27] for the policy π and for the impulse policy g. whose action set consists of two actions (intervene or do not intervene) we used a soft actor critic (SAC) process [14] LICRA is plug & play framework which enables these RL components to be replaced with any
177
178
180
182
RL algorithm of choice.
Convergence and Optimality of LICRA
189
A key aspect of our framework is the presence of two RL processes that make decisions in sequential order. In order to determine when to act the policy must learn the states to allow the policy x to perform an action which the policy must learn to select optimal actions whenever it is allowed to execute an action.
185
186
187
In this section, we prove that LICRA converges to an optimal solution of the system. Central to LICRA is a Q-learning type method which is adapted to handle RL settings in which the agent must also learn when to act. We then extend the result to allow for (linear) function approximators. We provide a result that shows the optimal intervention times are characterised by an 'obstacle condition' which can be evaluated online therefore allowing the g.
189
191
192
195
Given a function Q S X A + R VTT, n² E and Vg. g', Veru E S, we de- fine the intervention operator M by MT. (Sty an,)) = R(s,, ar.) - c(STu) ara) +
194
7 Eves P(s'; O s)ua (d') aT, ~ (-1887), ). where T& an intervention time.
195
196
The interpretation of M is the following: suppose that the agent is using the policy and at time Tk the system at state Syz and the agent performs an action ar ~ T(-(s=)). ). A cost of c(S ard) is then incurred by the agent and the system transitions to s' ~ P(: STT 87,). Lastly, recall v5.0 is the agent value function under the policy pair (#,g). Therefore, the quantity MQ measures the expected future stream of rewards after an immediate intervention minus the cost of intervention.
197
198
199
200
201
This object plays a crucial role in the LICRA framework which as we later discuss, exploits the cost structure of the problem to determine when the agent should perform intervention.
203
Given function u S + R, we define the Bellman operator T, by:
204
To T == max (s,,), +
Σ P(s):0,s) EES
E S.
206
The Bellman operator captures the nested sequential structure of the LICRA algorithm. In particular, the structure in (3) consists of an inner structure which consists of two terms: the first term is the expected future return given an action taken at the current state under the policy æ. The second term is the expected future return given no action is taken at the current state. Lastly, the outer structure is an optimisation which compares the expected return of the two possibilities and selects the maximum.
207
208
209
210
211
Our first result proves T is a contraction operator in particular, the following bound holds:
212
Lemma
The Bellman operator T a contraction, that is the following bound holds:
TTv Te'l S 1 --ll--1111. v'11,
5

---

213
where v, o' are elements of a finite normed vector space. We can now state our first main result:
Theorem Given any v= S A + R, the optimal value function is given by lim T k-+00
214
215 max u vx' where (T",9") is the optimal policy pair. R.DEII
The result of Theorem 1 enables the solution to the agent's 's impulse control problem to be determined using a value iteration procedure. Moreover, Theorem I enables a Q-learning approach [6] for finding the solution to the agent's 's problem.
217
219
Theorem Consider the following Q learrnng variant:
Q441($),a) Qe(si, at)
+ tay(st, an) max M ai), R(su, 0) ymax Qe(s)42, - (4)
d'EA
220
then Qi converges to Q* with probability 1, where st, S1+1 E S and as E A.
We now extend the result to (linear) function approximators:
222
Theorem Given a set of linearly independent basis functions = {0r,-. .............................. Φo with on E L2, Vk.
LICRA converges to limit point r* E IP which is the unique solution (rr) = Ør where
Fe = R + yP max (Mu, Moreover, r* satisfies: || Q* S (1 12/ I Q
225
The theorem establishes the convergence of LICRA a stable point with the use of linear function approximators. The second statement bounds the proximity of the convergence point by the smallest approximation error that can be achieved given the choice of basis functions.
226
227
228
Having constructed a procedure to find the optimal agent's 's optimal value function, we now seek to determine the conditions when intervention should be performed. Let denote by {Tx (== the points at which the agent decides to act or intervention times, so for example if the agent chooses to perform an action at state se and again at state sg, then 7Ι 6 and T2 The following result characterises the optimal intervention policy g and the optimal times [Tk30
230
231
232
The policy g is given by: g(st) HMM QF at); Vsa S, where Q is the solution in Theorem .M is the intervention operator and H is the Heaviside function, moreover the intervention times are ты inf Tk- M 3).8).
Proposition 1
234
236
Prop. I characterises the (categorical) distribution g. Moreover, given the function Q, the times {Tu} can be determined by evaluating if MQ Q holds.
key aspect of Prop. 238
that exploits the cost structure of the problem to determine when the agent should perform an intervention. In particular, the equality MQ Q implies that performing an action and incurring a cost for doing so optimal.
239
240
Budget Augmented LICRA via State Augmentation
241
242
We now tackle the problem of RL with a budget. To do this, we combine the above impulse control technology with state augmentation technique proposed in [28] The mathematical formulation of the problem is now given by the following for any 8 E S:
244
245
max U ( L. TE
2 0,
(S)
Th
246
where n N is a fixed value that represents the maximum number of allowed interventions and LAS1 5% is equal to one if an impulse was applied at time i and zero if it was not. In order to avoid dealing with a constrained MDP, we propose to introduce new variable z tracking the remaining
247
248
number of impulses: 2 = E Σx21 5%.- We treat another state and augment the
state-space resulting in the transition P:
250
251
8t-1 P(((s,,)))
2411
20
(6)

---

R(st, ai)
2 2 0,
252 In order to avoid violations, we reshape the reward as follows: R(s,, =t, az) =
--
24 < 0,
253 where > O is large enough hyper-parameter ensuring that there are no safety violations. To summarise we aim to solve the following problem:
T. (s, 2)=
R(se Re, ael ~ at les mt)
256
where the policy now depends on the variable 25. Note that P in Equation 6 is a Markov process and, the rewards R are bounded, as long as the rewards R are bounded. Therefore, we can apply directly the results for impulse control to this case well. We denote the augmented MDP by
257
(S x 2, A.A.P,R R), where the space of the augmented state. We have the following.
259
Theorem Consider the MDP M for the problem 7, then:
260
a) The Bellman equation holds, i.e. there exists function ū..5.8 s.th. 8-. 2)
(2),)))))) where the optimal policy for M has the form max [R(s,2, 2, a) + x / (T 2'))), aeA 2));
282
b) Given T: Sx 2 R. the stable point solution for M a given by lim T
max R
264
k-yop
#
T R where (T* ,,",0" g*) is an optimal policy of M and T is the Bellman operator of M.
285
286
The result has several important implications. The first that we can use modified version of LICRA to obtain the solution of the problem while guaranteeing convergence (under standard assumptions). Secondly, our state augmentation procedure admits a Markovian representation of the optimal policy.
267
288
Experiments
269
We will now study empirically the performance of the LICRA framework. In experiments, we use different instances of LICRA, one where both policies are trained using PPO update (referred to LICRA PPO) and one where the policy deciding whether to act is trained using SAC and the other policy trained with PPO (referred to as LICRA (AC). We have benchmarked both these environments, where it would be natural to introduce the concept of the cost associated with actions. We lastly performed series of ablation studies which
271
273
algorithms together with common baselines
275
test LICRA's ability to handle different cost functions including the case when c(s, a) which
defer to the Appendix which also contains further experiment details.
277
-20
dlgorttee Boro, _500
-30
1po
100000
200000
300000
400000
500000
Imasttop
Figure 1: Training results in Merton investment problem for PPO style algorithms.
278
Merton's Portfolio Problem with Transaction Costs. Merton Investment Problem in which the
279
investor faces transaction costs [10] a well-known problem within finance. In our environment, the
280
agent can decide to move its wealth between a risky asset and a risk-free asset. The agent receives a reward only at the final step, equal the utility of the portfolio with risk aversion factor equal to
282
0.5. If the final wealth of risky asset is sT and final wealth of risk-free asset is er, then the agent will
283
receive a reward of u(z) 2/ST ST + Cr. The wealth evolves according the following SDE:
dW, = (r + (r W, + W.p.odB,
(8)
7

---

2.5
LOKA
Average Epicode Return
2.0
PPO
Lagranggn
Lago-nagnannian
0.5
0.0
00.5
-1.0
0.0
0.2
0.4
0.6
0.8
1.0
Training Steps (mill)
(a)
(b)
Figure 2: a) Drive Environment. b) Training results in drive environment.
where Wt is the current wealth and the state variable, dB, is increment of Brownian motion and p is the proportion of wealth invested in the risky asset. We set the risk-free return r = 0.01, risky asset return μ 0.05 and volatility a We discretise the action space so that each step the agent has three actions available: move 10% of risky asset wealth to the risk-free asset, move 10% of risk- free
285
286
287
288
asset wealth the risky asset or do nothing. Each time the agent moves the assets, incurs cost
289
of 1 i.e. transaction fee. The agent can act after time interval of 0.01 seconds and the episode ends after 75 steps. The results of training are shown in Fig. which clearly demonstrates that LICRA PPO finds a better policy than standard PPO. Also comparing the variance among different seeds, we can see that LICRA PPO a much more stable algorithm than the other two.
290
291
292
Driving Environment Fuel Rationing. We studied an autonomous driving scenario where fuel- efficient driving a priority. One of the main components of fuel-efficient driving is controlled usage of acceleration and braking, in the sense that I) the amount of acceleration and braking should be limited 2) accelerations should be performed slowly and gently. We believe this a problem where LICRA should thrive the impulse control agent can learn to restrict the amount of acceleration and braking in the presence of other cars and choose when to allow the car to decelerate naturally. [17] environment vehicle is our controlled vehicle and the goal to avoid crashing into other vehicles whilst driving the continuous acceleration action, C(a.)= = K + af. where K >> 0 is fixed constant cost of taking any action, and de [-1, 1], with larger values of acceleration braking being penalised more. The results presented in Fig. (2.b). Notably, LICRA is able to massively outperform the baselines, especially our
294
296
298
We used the highway-
highway task (see Fig (2. a)) where the green
300
at reasonable speed. We add cost function into the reward term dependent
302
304
305
safety specific baselines which struggle to deal with the cost function associated with the environment.
306
We believe one reason for the success of LICRA is that it is far easier for it to utilise the null action
307
of zero acceleration/braking than the other algorithms, whilst all the algorithms have guaranteed
308
cost at every time step whilst not gaining a sizeable reward to counter the cost.
100
50
Episode
-50
UICAA SAC
PPO
-100
SAC
PPO Pro-Lagrangian
TRPO- www.e-igggrgggg. gian
Training Stops (mll)
(a)
(b)
Figure 3: a) The lander must land on the pad between two flags. b) Training results Lunar Lander. Lunar Lander Environment. We tested the ability of LICRA to perform in environment that simulate real-world physical dynamics. We tested LICRA's performance the Lunar Lander environment in OpenAl gym [7] which adjusted to incorporate minimal bounded costs in the reward definition. In this environment, the agent is required to maintain both a good posture mid-air air and reach the landing pad as quickly as possible. The reward function is given by:
Reward (st) = 3 (1 1d, -d== 3 * (1 10, - (1 1w, -ww-ww--st -0.03 * FuelSpent(s;) 10 (V: v.-1) 10 (wr wi-1) + 100 hasLanded (st)
8

---

309
where di is the distance to the landing pad, is the velocity of the agent, and wi is the angular velocity of the agent at time t. 1x is the indicator function of taking actions, which is 1 when the statement X is true and 0 when X is false. Considering the limited fuel budget, we assume that we have a fixed cost for each action taken by the agent here, and doing nothing brings no cost. Then, to describe the goal of the game, we define the function of the final status by haslLandedO. which is
310
312
313
314
0 when not landing; when the agent has landed softly on the landing pad: and -1 when the lander runs out of fuel or loses contact with the pad on landing. The reward function rewards the agent for reducing its distance to the landing pad, decreasing its speed to land smoothly and keeping the angular speed minimum prevent rolling. Additionally, it penalises the agent for running out of fuel and deters the agent from taking off again after landing.
316
318
319 By introducing reward function with minimally bounded costs, our goal was to test if LICRA can exploit the optimal policy. In Fig. 3, we observe that the LICRA agent outperforms all the baselines, both in terms of sample efficiency and average test return (total rewards at each timestep). We also observe that LICRA enables more stable training than PPO, PPO-Lagrangian and CPO.
320
322
Ablation Study 1. Prioritisation of Most Important Actions. We next tested LICRA's ability to prioritise where performs actions when the necessity to act varies significantly between states. To
324
test this, we modified the Drive Environment to now consist of a single lane, start state and goal state start (at the end) where there is a reward. With no acceleration, the vehicle decreases velocity.
326
To reach the goal, the agent must apply an acceleration a E [-1, 1]. Each acceleration o, incurs
328 a cost C(a)) as defined above. At zones = 1, 2,3 3 of the lane, if the vehicle is travelling below a velocity Umin it is penalised by a strictly negative cost C₁ where e₁ C2 C3. As shown in Fig. 4, when the intervention cost increases i.e. when K + co, LICRA successfully prioritises the highest penalty zones to avoid incurring large costs.
330
0.01
K. 0.2
1.2
06954923
LO241C
-0.4
6-1
644
822
Vicatiie COARt
Vilition
Vilittinn Court
Figure 4: Results for Ablation Study 1. Heatmaps display the number of times the agent drives below Umin in the penalty zones. Violation I refers to the lowest cost zone, whilst Violation 3 refers to the largest cost zone. K refers to the fixed cost for taking an action.
8 Conclusion
332
333
We presented a novel method to tackle the problem of learning how to select when to act in addition to learning which actions to execute. Our framework, which is a general tool for tackling problems of this kind seamlessly adopts RL algorithms enabling them to efficiently tackle problems in which the agent must be selective about when it executes actions. This is of fundamental importance in practical settings where performing many actions over the horizon can lead to costs and undermine the service life of machinery. We demonstrated that our solution, LICRA which at its core has a sequential decision structure that first decides whether or not an action ought to be taken under the
334
335
336
337
338
339
action policy can solve tasks where the agent faces costs with extreme efficiency as compared to leading reinforcement learning methods. In some tasks, we showed that LICRA able to solve problems that are unsolvable using current reinforcement learning machinery. We envisge that this framework can serve as the basis extensions to different settings including adversarial training for solving variety of problems within RL.
341
342
343
References
346
[1] Ali Ajdari, Maximilian Niyazi, Nils Henrik Nicolay, Christian Thieke, Robert Jeraj, and Thomas Bortfeld. Towards optimal stopping in radiation therapy. Radiotherapy and Oncology, 134:96-

---

348
100, 2019.
[2] Luluwah Al-Fagih. The british knock-out put option. International Journal of Theoretical and Applied Finance, 18(02):1550088, 2015.
350
351
[3] Juan J Alcaraz, Jose A Ayala-Romero, Javier Vales-Alonso, and Fernando Losilla-López. Online reinforcement learning for adaptive interference coordination. Transactions on Emerging Telecommuunications Technologies, 31(10)xe0877, 2020.
352
[4] Sebastian Becker, Patrick Cheridito, and Amulf Jentzen. Deep optimal stopping. arXiv preprint 1rXiv:1804.05394 2018.
355
356
[5] Albert Benveniste, Michel Métivier, and Pierre Priouret. Adaptive algorithms and stochastic approximations, volume 22. Springer Science & Business Media, 2012.
357
358
[6] Dimitri P "errtsekas. Approximate dynamic programming. Athena scientific Belmont, 2012.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.015500 2016.
360
381
[8] Andrew S Caplin and Daniel F Spulber. Menu costs and the neutrality of money. The Quarterly Journal of Economics, 102(4):703-72) 1987.
362
363
99] Shuhang Chen, Adithya M Devraj, Ana Bušić, and Sean Meyn. Zap q-learning for optimal stopping. In 2020 American Control Conference (ACC), pages
IEEE, 2020.
365
[10]
Mark HA Davis and Andrew Norman. Portfolio selection with transaction costs. Mathematics of operations research, 15(4):676-77) 1990.
366
[11] Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465 472.. Citeseer, 2011.
368
370
[12] Abderrahim Fathan and Erick Delage. Deep reinforcement learning for optimal stopping with application in financial engineering. arXiv preprint arXiv:2105.08877, 2021.
372
[13] F Grandt Jr. Damage tolerant design and nondestructive inspection-keys aircraft airworthiness. Procedia Engineering, 17:236-244 246, 2011.
373
[14] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905. 2018.
375
377
[15] Tommi Jaakkola, Michael Jordan, and Satinder P Singh. Convergence of stochastic iterative dynamic programming algorithms. In Advances in neural information processing systems, pages 703-710, 1994.
378
379
380
[16] Ralf Korn. Some applications of impulse control in mathematical finance. Mathematical Methods of Operations Research, 50(3):493-55 518., 1999.
382
[17] Edouard Leurent. An environment for autonomous driving decision-making https://github. com/ www.reet/thgghaaaeee. env, 2018
383
[18] Harry M Markowitz. Foundations of portfolio theory. The journal finance, 46(2):469-47) 1991.
385
386
[19] David Mguni. Duopoly investment problems with minimally bounded adjustment costs. arXiv
387
preprint arXiv: /9944, 2018
388
[00] David Mguni. Optimal capital injections with the risk of ruin: A stochastic differential game of impulse control and stopping approach. arXiv preprint 105.01.88, 2018
389
390
[21] David Mguni. Optimal selection of transaction costs dynamic principal-agent problem. arXiv preprint wtXiv:1805000066 2018.
10

---

382
[22] David Mguni. A viscosity approach to stochastic differential games of control and stopping involving impulsive control. arXiv preprint arXiv:1803. 11432, 2018.
393
394
[23] David Mguni. Cutting your losses: Learning fault-tolerant control and optimal stopping under adverse risk. arXiv preprint arXiv: 1900:0902..00445, 2019.
[4]
Bernt Øksendal. Stochastic differential equations. In Stochastic differential equations, pages 65-84. 84. Springer, 2003.
397
398
25
Bernt Øksendal and Agnès Sulem. Approximating impulse control by iterated optimal stopping. In Applied Stochastic Control of Jump Diffusions, pages 255-272. Springer, 2019.
399
400
[26]
Jing-Chenn Cheng Pang, Tian Xu, Sheng Jiang, Yu-Ren Liu, and Yang Yu. Sparsity prior regularized q-learning for sparse action tasks. arXiv preprint arXiv:210S. 08666, 2021.
27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707.70077747 06347, 2017.
403
404
[28] Aivar Sootla, Alexander I. Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David Mguni, Jun Wang, and Haitham Bou-Ammar. Sauté RL: Almost surely safe reinforcement learning using state augmentation. arXiv preprint arXiv:2202.06558, 2022.
406
407
[29] Aivar Sootla, Diego Oyarzún, David Angeli, and Guy-Bart Stan. Shaping pulses to control bistable systems: Analysis, computation and counterexamples. Automatica, 63:254-264, 2016.
408
409
[30] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
[31] John N Tsitsiklis and Benjamin Van Roy. Optimal stopping of markov processes: Hilbert space theory, approximation algorithms, and an application pricing high-dimensional financial derivatives. IEEE Transactions on Automatic Control, 44(10): 180::1448-1555 1999.
412
[32] Pu Zhao, Yanzhi Wang. Naehyuck Chang, Qi Zhu, and Xue Lin. A deep reinforcement learning framework for optimizing fuel economy of hybrid electric vehicles. In 2018 23rd Asia and South Pacific design aulomation conference (ASP-DAC), pages 196-202. IEEE, 2018.
415


Reviews:
[{"review_id": "LbAoQyNLfw", "reviewer": "Reviewer_UN8M", "summary": "This paper studies a class of sequential decision making problems, in which taking an action can incur a large cost. To address these problems, they proposed a method named Learnable Impulse Control Reinforcement Algorithm (LICRA), which learns both when to act and which action to take. For the proposed algorithm, they introduced a new Bellman operator and established convergence results. Then they generalized LICRA with state augmentation to solve the RL problem with a budget. Finally, they tested LICRL on three tasks.  ", "questions": "1. The operator of $\\mathcal{M}^{\\pi, g}$. This operator should be applied on **any** function in $S \\times A \\rightarrow \\mathbb{R}$. Why use the notation $Q^{\\pi, g}$, which depends on $(\\pi, g)$? Besides, the function $\\mathcal{M}^{\\pi, g} Q^{\\pi, g}$ should be dependent on $Q^{\\pi, g}$. However, according to the definition given in line 186, $\\mathcal{M}^{\\pi, g} Q^{\\pi, g}$ is independent of $Q^{\\pi, g}$. Besides, in line 191, what does the notation $\\mathcal{M} v^{\\pi, g}$ mean?\n2. The operator T in line 195. The operator T does not depend on any policy $(\\pi, g)$. However, in its definition, the terms $\\mathcal{M}^{\\pi, g}$, $Q^{\\pi, g}$ and $v^{\\pi, g}$ all depend on some policy $(\\pi, g)$, \n", "limitations": "The authors do not mention any limitations or potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. They proposed a reasonable LICRA algorithm that utilizes two separate policies to learn when to act and which action to take. It was shown that LICRA can also solve the RL problem with a budget.\n\n\nWeakness:\n1. The proposed method LICRA is not well-motivated and it is unclear why we need to learn two separate policies. As mentioned by the authors, to address RL problems in which taking an action can incur a cost, we can simply augment the action space with a null action and apply classical RL algorithms. Compared with this simple method, LICRA has an identical search space (action space) but requires twice the computational cost for learning the additional actor and critic. Besides, constrained RL approaches can also be applied in this setting. The authors do not show the advantages of LICRA over the methods mentioned before.\n2. The empirical evaluations are weak. First, the algorithms for comparison are too few. The constrained RL algorithms should be involved for comparison. Second, the improvements of LICRA are not clear on Merton’s Portfolio Problem and Lunar Lander task. Third, the introductions of the baseline algorithms in Figure 2 and Figure 3 are missing.\n3. The theoretical results and notations in Sec 5 are misleading. See the question part for details. I encourage the authors to re-write this part to improve the clarity.  \n4. This paper studies RL problems where taking an action can incur a large cost. A closely related work [1] is missing. [1] studies the RL problems where the amounts of some specific actions are limited, which is similar to the problems studied in this paper.\n\nReferences:\n[1]. Ji-Cheng, Pang, et al. \"Sparsity Prior Regularized Q-learning for Sparse Action Tasks.”arXiv preprint, arXiv: 2105.08666, (2021).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "6Pf0dtqzB4i", "reviewer": "Reviewer_A4kA", "summary": "This paper proposes an algorithm for finding the optimal policy in sequential decision problems where there is a cost associated with taking an action. At each timestep, the agent can choose to perform an action or do nothing. LICRA learns two RL policies: one for deciding when to act, and another for taking actions. By first learning when to act, the complexity of the problem is reduced compared to standard RL configurations. ", "questions": "* In Figure 1, what are the measures of centrality and spread for the different algorithms? How many random seeds were used? How much hyperparameter tuning was necessary to achieve the results shown. Same questions for Figures 2 and 3.\n* Please provide brief descriptions of each baseline algorithm used, and why they were chosen to be compared with LICRA.\n* What do minibatches refer to in Appendix 12?\n* What do optimization epochs refer to in Appendix 12?\n* In Appendix 12, there are no hyperparameters in square brackets following the notice in the second sentence. This section seems very incomplete to me. The SAC and PPO variants of LICRA should have different sets of hyperparameters. Are these hyperparameters also used for the baseline algorithms? Are these hyperparameters the same for both $\\pi$ and $g$?\n* Are the costs included in the Average Episodic Return reported in the figures? It could be insightful to compare the baselines using the vanilla reward.", "limitations": "I think the potential limitations of LICRA should be discussed in more detail. The experimental environments chosen are very well suited to LICRA since they involve the agent positioning itself in a certain configuration (e.g., spaceship over the goal in lunar lander) and maintaining that configuration by not acting. It would be nice to see how LICRA performs in environments that are less suited to it. For example, what about tasks where the agent should be acting “most” of the time, making it harder for $g$ to learn the few states where it should intervene?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Originality\nThe authors present a novel method for impulse control using two RL policies.\n\n#### Quality\nThe paper does a nice job of showcasing practical applications of LICRA in environments such as finance and autonomous driving. I did not examine the proofs in the appendix very closely, but the results seem to follow. \n\nI think the related work section is lacking descriptions and citations for related problem settings such as RL safety and hierarchical RL. Many of these works use a safety policy or high-level policy to intervene the lower-level policy during training. Since LICRA trains two RL processes, it would be interesting to see how it compares to other methods that do the same.\n\n#### Clarity\nThe paper is well structured and easy to understand. Algorithm 1 presents a procedure that should be easy to implement with standard auto-differentiation libraries. During the exposition of LICRA, the authors state that adding the zero action is not a good solution to impulse control. I think a small motivational example could be good for demonstrating this. \n\n##### Nits\n* The last paragraph of the introduction should be rewritten for grammar and clarity\n* Line 28: result \"in\" catastrophic losses\n* Line 56: series of results\n* Line 82: form of\n* Line 90: task is to\n* Line 135: not significantly impact\n* Line 190: $P(\\cdot ;  a_{\\tau_k}, s_{\\tau_k})$\n\n#### Significance\nThe results are useful for applications involving impulse control. Researchers or practitioners may use LICRA due to its effectiveness and how easy it is to implement in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5XnvuxafN86", "reviewer": "Reviewer_wRMk", "summary": "According to the authors, this paper studies the problem of choosing when to act, also known as the impulse control problem. Choosing to abstain from actions has relevance in various real-world scenarios where doing nothing can be the right decision. As the authors discuss, this can be because of some costs associated with taking action. A motivating example of their work is portfolio management, where not buying or selling financial instruments is the right move as fees are associated with each action.\n\nThe authors model propose a framework that incorporates the impulse control problem in the reinforcement learning problem. They accomplish this by representing costs associated with actions taken in the reward function. They decompose the agent policy into two policies: the original action space policy and a binary policy to choose whether to act or not. They argue against including this action in the rest of the action set because it would be inefficient in the learning process due to action cardinality. \n\nThe authors further prove theoretical results that their approach will converge to the optimal Q function. They do this in the tabular and linear function approximator cases. Their experiments further validate the framework, and the authors suggest their framework appropriately deals with these challenges empirically compared to baselines. \n", "questions": "- Are features shared for both policies in the experiments or are they entirely separate?\n- Can the action decision policy be a separate linear head trained with a different reward signal?\n-  Were any experiments done to validate the theoretical results?\n- A statement in section 4 seems important to evaluate: \"if the ratio of s to s_i is large, learning g before pi is essential\". What was the rationale for abstaining from testing this scenario? \n- In theorem 2, the max is between taking action or future expected reward after doing nothing. In terms of having taken action, why is it a_t and not max_{a_t}? Wouldnt this make this make it more of a SARSA update?\n- What is delta in section 6 Equation 4? I could not find the definition.\n- What is the state variable of the Merton problem? Does it have any sequential relations?\n- In the Merton experiments, are the results sensitive to the environment parameters of the SDE? I.e., is LICRA robust to these choices?\n- Which hyperparameters were tuned over? It is mentioned in the 12 Hyperparameter section that \"square brackets indicate ranges,\" but I did not see any in the hyperparameter in the table using square brackets.\n- Equation 14 of proof of theorem 1, inconsistent usage of lambda (is its function of 1 or 2 inputs?). What does \"I(\\tau)\" mean in Equation 14?\n- Please tighten up notation in theorem one proof; the proof starts with \\lambda, which becomes \\phi and then \\theta. It is not unclear why these variables change.\n- A paper that might be related is the following: Nonparametric learning for impulse control problems Soren Christensen and Claudia Straugh. Disclaimer: I thoroughly read the paper, but they mention impulse control and reinforcement learning \n- Would it be possible to include in the preliminary discussion on Impulse Control Problem? This seems central to your research but is omitted in the paper.\n- Consider restructuring your existing experiments section. It feels like Figures 1, 2, and 3 are, in some sense, experiments of similar conclusions (i.e., our method performs well). The ablations you do in the appendix seem valuable to include in the main paper as it isolates the effects of hyperparameters of the framework.\n- Consider moving the less crucial details of the Merton problem to the appendix. For example, the SDE feels unnecessary in the main paper. \n- You reference Fig. 5 in line 462 when it seems you mean Figure 4. \n- Consider removing the examples in \"3 Preliminaries\". They do not add much to the discussion that was not already mentioned in the introduction. \n- Consider using a more digestible metric for performance over learning curves (e.g. Final average performance). It’s a bit difficult to distinguish the performance gains of your algorithm.\n- Include Compute resources used. This is mentioned as required in the Checklist, but these details weren’t mentioned in the appendix or paper.\n- Figure 6: If you change to a single metric (average final reward, the area under the curve, success rate, etc.) You could consider using a radar plot to visualize performance. As higher is assumed to be better, you could have the rewards on the outside, and each radar is your algorithm. The bigger the circle, the better your algorithm is across cost functions.\n- What is Figure 5 showing exactly? As K goes up, fewer violations should occur as it's terrible to decelerate in these zones, but it wasn't clear. It could help to visualize this experiment to explain it. \n", "limitations": "It's hard to gauge the limitations of their work because the authors do not seem to discuss this explicitly in the paper. In terms of negative societal impact, there doesn't seem to be any apparent reason their work could have negative social impacts. ", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "According to the authors, the paper presents a significant problem that has not been studied in the reinforcement learning problem. From the reviewer's experience in robotics, it is a valid consideration that not sending a new action can be the optimal decision. The author's theoretical results support the validity of their proposed algorithms. The fact that their algorithm can augment existing reinforcement learning algorithms is a great advantage as it means one can quickly implement the additional policy for real-world applications. The way the authors structured their problems seems appropriate to understand the problem they are interested in addressing. \n\n\nHowever, a concern with this paper is with the experiment evaluations. Much more evaluation is necessary to validate the LICRA framework. A few crucial things that come to mind would be to validate the theoretical results in the tabular and linear cases. The authors discuss Q-learning-like algorithms, but it seems like they do not even propose it by itself as a new algorithm or do any experiments with it. Not evaluating this update rule feels like a missed opportunity to isolate and understand the influence of the LICRA framework in more easily reproducible settings.\n\nSupposing the there were reasonable justification against these types of experiments, their existing experiments could still be improved. One discrepancy in the experiments is the different versions of LICRA algorithms that the authors claim to study instead of report. In the Experiments introduction, they introduce LICRA_PPO and LICRA_SAC, but in Figure 1, LICRA_SAC is not present; in Figure 2, there is just \"LICRA,\"; and only in Figure 3 are both of these algorithms present. Perhaps this is nit-picking, but from the author's inclusion of different combinations of RL policies with their algorithm, one would expect full coverage of the combinations (so four different versions of LICRA if using both SAC and PPO for each sub-policy). Additional comments are left for the question section as they are more about details to be included and for clarification. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject