Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints
Authors: David Henry Mguni
Aivar Sootla
Juliusz Krzysztof Ziomek
Oliver Slumbers
Zipeng Dai
Kun Shao
Jun Wang
Venue: ICLR 2023 poster (2023)
Keywords: dynamic programming
impulse control
optimal stopping
reinforcement learning

Abstract: Many real-world settings involve costs for performing actions; transaction costs
in financial systems and fuel costs being common examples. In these settings,
performing actions at each time step quickly accumulates costs leading to vastly
suboptimal outcomes. Additionally, repeatedly acting produces wear and tear and
ultimately, damage. Determining when to act is crucial for achieving successful
outcomes and yet, the challenge of efficiently learning to behave optimally when
actions incur minimally bounded costs remains unresolved. In this paper, we intro-
duce a reinforcement learning (RL) framework named Learnable Impulse Control
Reinforcement Algorithm (LICRA), for learning to optimally select both when
to act and which actions to take when actions incur costs. At the core of LICRA
is a nested structure that combines RL and a form of policy known as impulse
control which learns to maximise objectives when actions incur costs. We prove
that LICRA, which seamlessly adopts any RL method, converges to policies that
optimally select when to perform actions and their optimal magnitudes. We then
augment LICRA to handle problems in which the agent can perform at most k < ∞
actions and more generally, faces a budget constraint. We show LICRA learns the
optimal value function and ensures budget constraints are satisfied almost surely.
We demonstrate empirically LICRA’s superior performance against benchmark
RL methods in OpenAI gym’s Lunar Lander and in Highway environments and a
variant of the Merton portfolio problem within finance.

Published as conference paper at ICLR 2023
TIMING IS EVERYTHING: LEARNING TO ACT SELEC- TIVELY WITH COSTLY ACTIONS AND CONSTRAINTS
David Mgunit.1, Aivar Sootla', Juliuss Ziomek!, Oliver Slumbers!, Zipeng Dai', Kun Shao', Jun Wangh. Huawei Noah's s Ark Lab PUCL, London, United Kingdom
ABSTRACT
Many real-world settings involve costs for performing actions; transaction costs in financial systems and fuel costs being common examples. In these settings, performing actions at each time step quickly accumulates costs leading to vastly suboptimal outcomes. Additionally, repeatedly acting produces wear and tear and ultimately, damage. Determining when to act is crucial for achieving successful outcomes and yet, the challenge of efficiently learning to behave optimally when actions incur minimally bounded costs remains unresolved. In this paper, we intro- duce reinforcement learning (RL) framework named Learnable Impulse Control Reinforcement Algorithm (LICRA), for learning to optimally select both when to act and which actions to take when actions incur costs. At the core of LICRA is a nested structure that combines RL and a form of policy known impulse control which learns to maximise objectives when actions incur costs. We prove that LICRA, which scamlessly adopts any RL method, converges to policies that optimally select when to perform actions and their optimal magnitudes. We then augment LICRA to handle problems in which the agent perform at most k actions and more generally, faces a budget constraint. We show LICRA learns the optimal value function and ensures budget constraints are satisfied almost surely. We demonstrate empirically LICRA's superior performance against benchmark RL methods in OpenAl gym's Lunar Lander and Highway environments and variant of the Merton portfolio problem within finance.
INTRODUCTION
There are many settings in which agents incur costs each time they perform an action. Transaction costs in financial settings (Mguni, 201 8a), fuel expenditure (Zhao et al., 2018), toxicity as side effect of controlling bacteria (Sootla al., 2016) and physical damage produced by repeated action that produces wear and tear just few among many examples (Grandt Jr, 2011). In these settings, performing actions at each time step is vastly suboptimal since acting in this way results in prohibitively high costs and undermines the service life of machinery. Minimising wear and tear is essential attribute to safeguard against failures that can result in catastrophic losses (Grandt Jr, 201 I).
Reinforcement learning (RL) is framework that enables autonomous agents to learn complex behaviours from interactions with the environment (Sutton & Barto, 2018). Within the standard RL paradigm, determining optimal actions involves making selection among many (possibly infinite) actions; a procedure that must be performed at each time-step as the agent decides on an action. In unknown settings, the agent cannot immediately exploit any topological structure of the action set. Consequently, learning not to take an action i.c performing a zero or null action, involves expensive optimisation procedures over the entire action set. Since this must be done each state, this process is vastly inefficient for learning optimal policies when the agent incurs costs for acting.
In this paper, we tackle this problem by developing an RL framework for finding both an optimal criterion to determine whether or not to execute actions as well as learning optimal actions. A key component of our framework is a novel combination RL with a form of policy known as impulse control (Mguni, 2018d;a). This enables the agent to determine the appropriate points to perform action as well as the optimal action itself. Despite its fundamental importance as a tool for tackling

---

Published as conference paper at ICLR 2023
decision problems with costly actions (Korn, 1999; Mitchell et al., 2014), presently, the use of impulse control within learning contexts (and unknown environments) is unaddressed.
We present RL impulse control framework called LICRA, which, our knowledge, is the first learning framework for impulse control. To enable learning optimal impulse control policies in unknown environments, we devise a framework that consists of separate RL components for learning when to act and, how to act, optimally. The resulting framework is a structured two-part Icarning process which differs from current RL protocols. In LICRA, at each time step, the agent firstly makes a decision whether to act or not leading to binary decision space (0, 1). The second decision part determines the best action to take. This generates subdivision of the state space into two regions; one in which the agent performs actions and another in which it does not act at all. We show the set of states where the agent ought to act is characterised using an casy-to-evaluate evaluate criterion on the value function and, LICRA quickly learns this set within which it then learns which actions optimal.
We then establish thcory that ensures convergence of Q-lcarning variant of LICRA to the optimal policy for such settings. To do this, we give series of results namely:
i) We establish dynamic programming principle (DPP) for impulse control and show that the optimal value function can be obtained foundation for an RL approach to impulse control.
limit of value iterative procedure (Theorem 1) which lays the
ii) We extend result i) to new variant of Q-learning which enables the impulse control problem be solved using our RL method (Theorem 2). Thereafter, we extend the result to (linear) function approximators cnabling the value function to be parameterised (Theorem 3).
iii) We characterise the optimal conditions for performing an action which we reveal to be simple "obstacle condition' involving the agent's value function (Proposition 1). Using this, the agent can quickly determine whether or not it should act and if so, then learn what the optimal action is. iv) In See. 6, we extend LICRA to include budgetary constraints so that each action draws from a fixed budget which the agent must stay within. Analogous to the development of i). we establish another DPP from which we derive a Q-learning variant for tackling impulse control with budgetary constraints (Theorem 4). A particular case of budget constraint is when the number of actions the agent can take over the horizon is capped.
Lastly, we perform set of experiments validate our theory within the Highway driving simulator and OpenAl's LunarLander (Brockman el al., 2016).
As we demonstrate in our experiments, LICRA learns to compute the optimal problems in which the agent faces costs for acting in an efficient way which outperforms leading RL baselines. Second, as demonstrated in Sec. 6, LICRA handles settings in which the agent has cap the total number of actions it is allowed to execute and more generally, generic budgetary constraints. LICRA is able to accommodate any RL algorithm unlike various RL methods designed to handle budgetary constraints.
2 RELATED WORK
In continuous-time optimal control theory (Øksendal, 2003), problems in which the agent faces a cost for each action are tackled with a form of policy known as impulse control (Mguni, 201 8d;a; Al-Fagih, 2015). In impulse control frameworks, the dynamics of the system are modified through a sequence of discrete actions or bursts chosen at times that the agent chooses to apply the control policy. This distinguishes impulse control models from classical decision methods in which agent takes actions at each time step while being tasked with the decision of only which action to take. Impulse control models represent appropriate modelling frameworks for financial environments with transaction costs, liquidity risks and economic environments in which players face fixed adjustment costs (e.g. memu costs) (Korn, 1999; Mguni, 2018b; Mundaca & Øksendal, 1998).
The current setting is intimately related to the optimal stopping problem which widely occurs in finance, economics and computer science (Mguni, 2019; Tsitsiklis & Van Roy, 1999). In the optimal stopping problem, the task is to determine a criterion that determines when to arrest the system and receive a terminal reward. In this case, standard RL methods are unsuitable since they require an expensive sweep (through the set of states) to determine the optimal point to arrest the system. The current problem can be viewed as an augmented problem of optimal stopping since the agent must now determine both sequence of points to perform action or intervene and their optimal magnitudes - only acting when the cost of action is justified (Øksendal & Sulem, 2019). Adapting

---

Published as conference paper at ICLR 2023
RL tackle optimal stopping problems has been widely studied (Tsitsiklis & Van Roy, 1999; Becker et al., 2018; Chen et al., 2020) and applied to a variety of real-world settings within finance (Fathan & Delage, 2021) and network operating systems (Alcaraz et al., 2020). Our work serves as natural extension of RL approaches to optimal stopping to the case in which the agent must decide at which points take series of actions. As with optimal stopping. standard RL methods cannot efficiently tackle this problem since determining whether to perform a O action requires a costly sweep through the action space at every state (Tsitsiklis & Van Roy, 1999). In (Pang et al., 2021) the authors introduce "sparse action" with a similar motivation as impulse control. However, the authors treat only the discrete action space case. Moreover, (Pang et al., 2021) do not provide a general theoretical framework of dealing with "sparse actions" but develop purely algorithmic solutions. Additionally, unlike the approach taken in (Pang et al., 2021), the problem setting we consider is one in which the agent faces cost for each action- this produces need for the agent to be selective about where it performs actions (but does not necessarily constrain the magnitude or choice of those actions).
Our framework bears resemblance to hierarchical RL (HRL) (Barto & Mahadevan, 2003). Policies in this hierarchy have usually different objectives. For example in sub-goal learning (Noelle, 2019), a top-level policy decides the goal of a low-level policy. In this case, the top-level policy gives control to the low-level policy for a number of time steps and/or until the sub-goal is achieved. In contrast, LICRA's top-level policy must make decision at every time step. One can note further similarities with specific HRL. framework - RL. with options (Precup et al., 1998; Klissarov & Precup. 2021). There the agent chooses an option and then selects an action (or a sequence of actions) based on the current state and the chosen option. Therefore, there are two policies that the agent learns. Options and HRL offer abstraction frameworks with higher levels of internal structure than most of the standard RL algorithms. This can significantly speed up Icarning in many situations. For example, in a setting with sparse rewards, a subgoal discovery algorithm can guide the agent to an optimal policy more quickly. Our impulse control framework offers even higher level of structure than options and HRL in order to determine when to act (Jinnai et al., 2019). This is achieved by exploiting the specificity of the impulse control setting which includes cost of acting while employing a structure that enables isolating null actions from the action space.
3 PRELIMINARIES
In RL, an agent sequentially selects actions to maximise its expected returns. The underlying problem is typically formalised as an MDP (S, A, P, R, 7) where S C IP is the set of states, C R is the (continuous) set of actions, P: S x A x + [0, 1] is transition probability function describing the system's 's dynamics, R: S x A R is the reward function mcasuring the agent's ' performance and the factor 7 E [0, 1) specifies the degree to which the agent's 's rewards are discounted over time (Sutton & Barto, 2018). At time E 0, 1,..., the system is in state 8, E S and the agent must choose an action at E AA which transitions the system to a new state 8t: reward R(sr, az). A policy т: : S x A + |0, 1J is a probability distribution over state-action pairs where π (als) represents the probability of selecting action a E AA in state s E S. The goal of an RL agent is to find a policy _ E II that maximises its expected returns given by the value function: en (s)
P(- (t, ae) and produces a
E E Y R(s,, az) a; ~ (-) so s) where II is the agent's 's policy set. The action value function is given by ((s, a) E R(s aa) = a, so s).
We consider a setting in which the agent faces at least some minimal cost for each action it performs. With this, the agent's 's task to maximise:
8o
(s) E
Σ
rt (R(st, at) C(st, C(st.a))) so
(I)
where for any state s E S and any action a E A, the functions R and C are given by R(s, a) R(s, a) LuE/oo + 0)(1 lueA/10}) where luEA/(O) is the indicator function which is 1 when E A/(0) and 0 otherwise and C(s,a)= := c(s, a) where positive (cost) function that introduces cost each time the agent performs an action. Examples of the cost function is quasi-linear function of the form c(st, u:) = K + s(a;) where f A -> R500 and x a positive real-valued constant. Since acting at each time step would incur prohibitively high costs, the agent must be selective when to perform an action. Therefore, in this setting, the agent's problem is augmented to learning both an optimal policy for its actions and, learning at which states to apply
S x A + R is a strictly

---

Published as conference paper at ICLR 2023
its action policy. Examples of problem settings of this kind include financial portfolio problems with transaction costs (Davis & Norman, 1990) where each investment incurs a fixed minimal cost and robotics where actions produce mechanical stress leading fatigue (Grandt Jr, 2011).
4 THE LICRA FRAMEWORK
In RL, the agent's problem involves learning to act at every state including those in which actions do not significantly impact on its total return. While we can add zero action the action set A and apply standard methods, we argue that this may not be the best solution in many situations. We argue the optimal policy has the following form:
7(-1s)
at
BES St: & Sr:
(2)
0
which implies that we simplify policy learning by determining the set Sy first the set where we actually need to learn the policy.
We now introduce a learning method for producing impulse controls which enables the agent to learn to select states to perform actions. Therefore, now agent is tasked with learning to act states that are most important for maximising its total return given the presence the cost for each action. To do this effectively, each state the agent first makes binary decision decide to perform
action. Therefore, LICRA consists of two core components: first, an RL process S x {0, + [0, 1] and second RL process S A [0, 1]. The role of g to determine whether or not action is be performed by the policy и at given state In LICRA, the policy π first proposes an action E A which observed by the policy g. activated, the policy и determines the action be selected. Therefore, prevents actions under for which the change in expected return does not exceed the costs incurred for taking such actions. While we consider now two policies n, g. cardinality of the action space does not change. Crucially, the agent must compute optimal actions only the subset of states chosen by g. By isolating the decision of whether to act or not, the LICRA framework may also reduce the computational complexity in this sctting.
Algorithm 1: Learnable Impulse Control Reinforcement Algorithm (LICRA)
1: Input: Stepsize a, batch size B, episodes K, steps per episode T, mini-epochs Initialise: Policy network (acting) m, Policy network (switching) g.
Critic network (acting VV, Critic network (switching V
3: Given reward objective function, R. initialise Rollout Buffers B,, Bg (use Replay Buffer for SAC)
4: for Nepisodes do
5: Reset state sp. Reset Rollout Buffers Br, Bg
6: for t 0, 1,... do
7:
Sample a x-1ss) Sample 91 ~ g(
8:
9:
if St then
10:
Apply a, so 8 ~ P(-|ar, St); Receive rewards Tı R(5,, at) c(5r, ai) Store (st, S1+1,T) in Br
12:
else
14:
Apply the null action so 8141 P(-10, Receive rewards Ti R(s,,)) 0).
16:
end if
17:
Store (5t. Do. S141.77 r() in B3
18:
end for
19:
// Learn the individual policies
20:
Update policy и and critic V, networks using B
21: Update policy g and critic Vg networks using B
end for
LICRA consists of two independent procedures: learning process for the policy v and simultane- ously, a learning process for the impulse policy which determines at which states to perform an

---

Published as conference paper at ICLR 2023
action. In Sec. 5, prove the convergence properties of LICRA. In our implementation, we used proximal policy optimisation (PPO) (Schulman et al., 2017) for the policy я and for the impulse policy g. whose action set consists of two actions (intervene or do not intervene) critic (SAC) process (Haarooj et al., 2018). LICRA is a plug & play framework which enables these RL components be replaced with any RL algorithm of choice. Above the pseudocode for LICRA, we provide full details of the code in Sec. 9 of the Appendix.
used soft actor
5 CONVERGENCE AND OPTIMALITY OF LICRA WITH Q-LEARNING
A key aspect of our framework is the presence of two RL processes that make decisions in sequential order. order to determine when to act the policy g must learn the states to allow the policy r to perform an action which the policy must learn lo select optimal actions whenever it is allowed to exccute an action. In this section, we prove that LICRA converges to an optimal solution of the system. Our proof is instantiated in Q-learning variant of LICRA which serves the natural basis for other extensions such as actor-critic methods. We then extend the result to allow for (linear) function approximators. We provide result that shows the optimal intervention times characterised by an "obstacle condition' which can be evaluated online therefore allowing the g policy to be computed online.
Given function Q S R, VT, T' E II and Vg. Vsr, E S, we define the intervention operator M by M (Q"" (ST,, a)] == Rsr, ) -c(s arz ) + Eves P(s'; On, s)Q" (s',, any ary))a la,, T (1187)) The interpretation M [Q is the following: suppose that at time т5 the system at state Sr, and the agent performs an immediate action az, (+)ss.)) A cost of c(5rg: Grg is then incurred by the agent and the system transitions to s' ~ P( Srx) whereafter the agent executes the policy (T,, g). Therefore M QT. is the expected future stream of rewards after an immediate action minus the cost of action. This object plays a erucial role in the LICRA framework which as we later discuss, exploits the cost structure of the problem to determine when the agent should take an action. Denote by M[Q the intervention operator acting on T7T when the immediate action is chosen according to an epsilon- policy. Given any 1 S - R. the Bellman operator T is:
To == max {MiQT (s) R(s,0) 0) + Σ P(s (() (s)))
E.
(3)
E'S
The Bellman operator captures the nested sequential structure of the LICRA method. In particular, the structure in (3) consists of an inner structure that consists of two terms: the first term M[QT.a is the estimated expected return given the current best estimate of the optimal action is executed and the policy æ is executed thereafter. The second term R(s, 0) + y Eves P(s', O, .. isolates the estimated expected return following no action and when the policy т is executed thereafter. Lastly, the outer structure is an optimisation which compares the two quantities and selects the maximum. Note that when is continuous (dense) set, this reduces the burden on an optimisation (possibly over function approximator) to exiract the point 0 in the action space when it is optimal not to act.
Our first result proves T' a contraction operator in particular, the following bound holds:
Lemma 1 The Bellman operator T is contraction, that is the following bound holds:
ITu Te'll S ||e v'l1,
where u of are elements of a finite normed vector space. We can now state our first main result:
Theorem Given any v :S x A + IR, the optimal value function is given by lim T n =
k-+00
max 9 = o0" where (x*, gg) is the optimal policy pair. R.GElI
The result of Theorem enables the solution to the agent's 's impulse control problem be determined using a value iteration procedure. Moreover, Theorem enables Q-learming approach (Bertsekas, 2012) for finding the solution to the agent's problem.
Note for the term MQQ the action input of the Q function is a dummy variable decided by the operator M.

---

Published as conference paper at ICLR 2023
Theorem Consider the following Q-learning variant: Q1+1(se, (5,,()))))) (t) Qe(st, an)
+ (st az) maxx M(Qr(se,, ae), R(se, 0) + + sup )1(
Qu(s) (4)
a'EA
then Qt converges to Q* with probability 1, where 8t, 8+11 E and at E A.
We now extend the result to (linear) function approximators:
Theorem 3 Given a set of linearly independent basis functions 4 = {ol,..... Op with oz E L3, Vk. LICRA converges limit point r* E R which is the unique solution to (r()rr 4r* where = R 7
Q} and r* satisfies: || Q*1l s (1 - IIQ Q*
The thcorem establishes the convergence of the LICRA Q-learning variant to a stable point with the use of linear function approximators. The second statement bounds the proximity of the convergence point by the smallest approximation error that can be achieved given the choice of basis functions.
Having constructed a procedure to find the optimal agent's 's optimal value function, we now seek to determine the conditions when intervention should be performed. Let us denote by {T=) )220 the points at which the agent decides to act or intervention times, so for example if the agent chooses perform an action state 56 and again state sg, then 71 6 and 72 8. The following result characterises the optimal intervention policy and the optimal times {Tu [TK}
The policy g is given by: g(s,) = [MM" --- ( at), Vs, E S, where Q is the solution in Theorem I, M is the intervention operator and H is the Heaviside function, moreover the intervention times are T₁ = inf >
Proposition 1
= 3)
Prop. 1 characterises the (categorical) distribution g. Moreover, given the function Q. the times {Tk} can be determined by evaluating if M[Q] Q holds. A key aspect of Prop. 1 is that it exploits the cost structure the problem determine when the agent should perform an intervention. In particular, the equality M[Q) Q implies that performing an action is optimal.
6 BUDGET AUGMENTED LICRA VIA STATE AUGMENTATION
We now tackle the problem of RL. with a budget. To do this, we combine the above impulse control technology with state augmentation technique proposed in (Sootla et al., 2022) The mathematical formulation of the problem is now given by the following for any 8 E S:
max (s) S. t. =E11,9
8 20
(5)
where N is fixed value that represents the maximum number of allowed interventions and Ens 5%_ is equal to one if an impulse was applied at time and zero it was not. In order to avoid dealing with a constrained MDP, we propose to introduce a new variable 2 tracking the remaining number of impulses: 2 = n Σ/D Ee>1 6%2- We treat 2N as another state and augment the state-space resulting in the transition P:
S111 a(-)s,, 211
n.
(6)
A21
To avoid violations, we reshape the reward as follows: R((s, 24, 41) = [ ar)
21 2 0,
1-
21 < 0,
where A > 0 is a large enough hyper-parameter ensuring there are no safety violations. summarise, we aim solve the following budgeted impulse control (BIC) problem nt : 2)
To
E 2 R(s,,, Zz, at) az (-1,,, #)} where the policy now depends on the variable 2,. Note
P. in Equation is a Markov process and, the rewards R. are bounded given the boundedness of R.
Therefore, we can apply directly the results for impulse control. We denote the augmented MDP by
M (S x Z, A, P. R, R),, where Z is the space the augmented state. We have the following:

---

Published as conference paper at ICLR 2023
Theorem Consider the MDP M for the BIC problem, then:
a) The Bellman equation holds, ie. 36. S x Z + R such that (7, max (R(s,2,u) + ylgg, x2- (6 22)),, where the optimal policy for M has the form asA
x (.)s,2)); b) Given a S x 2 + R, the stable point solution for M is given by lim Thamai =
k-000
max J " where (T is an optimal policy and T is the Bellman operator of M. #
The result has several important implications. The first is that we a modified version of LICRA to obtain the solution of the problem while guaranteeing convergence (under standard assumptions). Secondly, our state augmentation procedure admits a Markovian representation of the optimal policy,
7 EXPERIMENTS
We will now study empirically the performance of the LICRA framework. In experiments, we different instances of LICRA, one where both policies are trained using PPO update (referred to as LICRA_PPES and one where the policy deciding whether to act is trained using SAC and the other policy trained with PPO (referred to as LICRA SAC). We have benchmarked both of these algorithms together with common baselines on environments, where would be natural introduce the concept of the cost associated with actions. We performed a series of ablation studies (see Appendix) which examine i) LICRA's ability to handle different cost functions including the case when c(s, a) = O. ii) the benefits of LICRA in settings with a varying intervention regions. iii) the performance of the LICRA Q-learning variant in a discrete state space environment. In the following experiments, all algorithms apply actions that are drawn from an action set which contains 0 action. When this action is executed it produces costs and exerts no influence on the transition dynamics. Merton's Portfolio Problem with Transaction Costs. To exemplify the problem in applied
Agerttere
3cra, 200
200
20
-30
Figure 1: Training results in Merton investment problem for PPO style algorithms. (Confidence intervals over 20 seeds). Average final performance was 26.40 for LICRA_PPO and 19.60 for PPO.
setting, we choose well-known problem in finance known as the Merton Investment Problem with transaction costs (Davis & Norman, 1990). In this simple environment, the agent can decide to move its wealth between risky asset and a risk-free asset. The agent receives a reward only at the final step, equal to the utility of the portfolio with a risk aversion factor equal to 0.5. If the final wealth of risky asset is sy and final wealth of risk-free asset is cr, then the agent will receive reward of uu (x)) = 2/8T + CT. The wealth evolves according to the following SDE:
dW, (r + p. - r)) + WepeadB,
where W₂ is the current wealth (the state variable), dB, is an increment of Brownian motion and pr the proportion of wealth invested in the risky asset. We set the risk-free return 0.01, risky asset return 0.05 and volatility o = 1. We discretise the action space so that at each step the agent has three actions available: move 10% of risky asset wealth to the risk-free asset, move 10% of risk-free asset wealth to the risky asset or do nothing. Each time the agent moves the assets, it incurs a cost i.e. transaction fee. The agent can act after time interval of 0.01 seconds and the episode ends after 75 steps. The results of training are shown in Fig. which clearly demonstrates that LICRA_ PPO learns faster than standard PPO.

---

Published as conference paper at ICLR 2023
5
UCBA-PO
Average Ep sode Return
PPO
1.5
PRO-Legrengan
Training Steps (mil)
(a)
(b)
Figure 2: a) Drive Environment. b) Training results in drive environment. (Confidence intervals over 5 seeds) Average final performance was 1.75 for LICRA SAC, 0.37 for SAC, 2.63 for LICRA_PPA 0.24 for PPO.
Driving Environment Fuel Rationing. We studied an autonomous driving scenario where fuel- efficient driving is priority. One of the main components of fuel-efficient driving is controlled usage of acceleration and braking, in the sense that 1) the amount of acceleration and braking should be limited 2) if accelerations should be slowly and gently. We believe this problem where LICRA should thrive as the impulse control agent can learn to restrict the amount of acceleration and braking in the presence of other cars and choose when allow the car decelerate naturally. We used the highway-en (Leurent, 2018) environment on a highway task (see Fig (2. a)) where the green vehicle our controlled vehicle and the goal is to avoid crashing into other vehicles whilst driving at reasonable speed. We add cost function into the reward term dependent on the continuous acceleration action, C(a,) K + af, where K > 0 is fixed constant cost of taking any action, and a E [-1, 1], with larger values of acceleration or braking being penalised more. The results are presented in Fig. (2.b). Notably, LICRA able to massively outperform the basclincs, especially our safcty specific basclines which struggle to dcal with the cost function associated with the environment. We believe one reason for the success LICRA is that is far easier for to utilise the null action of zero acceleration/braking than the other algorithms, whilst all the algorithms have guaranteed cost at every time step whilst not gaining a sizeable reward to counter the cost.
Aveerage Episode Reruum
50
0
LICRA PDD
100
150
PAO Lagrangian TRPO-Lopranglen
0.0
0.2
0.4
U.S
1.0
Training Steps (mill)
Figure 3: a) The lander must land on the pad between two flags. b) Training results in Lunar Lander. Note that we use 6 different random seeds to get the error bars in this figure. The average final performance was 98.03 for LICRA SAC, 61.20 for SAC, 58 84 for (CRAA_POO 49.77 for PPO.
Lunar Lander Environment. We tested the ability of LICRA perform in environment that simulate real-world physical dynamics. We tested LICRA's performance the Lunar Lander environment OpenAl gym (Brockman et al., 2016) which we adjusted incorporate minimal bounded costs in the reward definition. In this environment, the agent is required to maintain both good posture mid-air and reach the landing pad as quickly as possible. The reward function given by:
Reward (sæ) 3 (1 1d, 10) (1 1 = 3 1 -.
-0.03 * FuelSpent(s:) 10 (vr vz-1) 10 (we w2-1) 100 hasLanded (st)
where d, is the distance to the landing pad, is the velocity of the agent, and wy is the angular velocity of the agent at time f. 1x the indicator function of taking actions, which 1 when the statement X is true and 0 when X is false. Considering the limited fuel budget, we assume that we have fixed cost for each action taken by the agent here, and doing nothing brings no cost. Then,

---

Published as conference paper at ICLR 2023
to describe the goal of the game, we define the function of the final status by hasLandedO,, which is O when not landing; when the agent has landed softly on the landing pad; and - when the lander runs out of fuel loses contact with the pad on landing. The reward function rewards the agent for reducing its distance to the landing pad, decreasing its speed to land smoothly and keeping the angular speed minimum prevent rolling. Additionally, it penalises the agent for running out of fuel and deters the agent from taking off again after landing.
By introducing reward function with minimally bounded costs, our goal was test if LICRA can exploit the optimal policy. In Fig. 3, we observe that the LICRA agent outperforms all the baselines, both in terms of sample efficiency and average test return (total rewards at each timestep). We also observe that LICRA enables more stable training than PPO, PPO-Lagrangian and CPO.
Ablation Study 1. Prioritisation of Most Important Actions. We next tested LICRA's ability to prioritise where performs actions when the necessity to act varies significantly between states. To test this, we modified the Drive Environment to now consist of a single lane, a start state and goal state start (at the end) where there reward. With acceleration, the vehicle decreases velocity. To reach the goal, the agent must apply an acceleration 04 E [-1, 1]. Each acceleration a incurs cost C(as) as defined above. At zones k 1, 2,3 3 of the lane, if the vehicle is travelling below a velocity Umin, it is penalised by strictly negative cost where c₁ < C2 < C3. As shown in Fig. 4, when the intervention cost increases i.e. when K oo, LICRA successfully prioritises the highest penalty zones to avoid incurring large costs.
0.01
93448
99957
-0
Werae 211
8.24-266
200422
-4
velionn
Maramian COURt
Valviii COUFE
Vacaran DALIE
Figure 4: Results for Ablation Study 1. Heatmaps display the number of times the agent drives below Umin in the penalty zones. Violation 1 refers to the lowest cost zone, whilst Violation 3 refers to the largest cost zone. K refers to the fixed cost for taking an action. Results averaged over 5 seeds.
8 CONCLUSION
We presented a novel method to tackle the problem of learning how to select when to act in addition to learning which actions to execute. Our framework, which is a general tool for tackling problems of this kind seamlessly adopts RL algorithms enabling them to efficiently tackle problems in which the agent must be selective about when it executes actions. This is of fundamental importance in practical settings where performing many actions over the horizon can lead to costs and undermine the service life of machinery. We demonstrated that solution, LICRA which at its core has sequential decision structure that first decides whether or not an action ought to be taken under the action policy can solve tasks where the agent faces costs with extreme efficiency as compared leading reinforcement learning methods. In some tasks, we showed that LICRA is able to solve problems that are unsolvable using current reinforcement learning machinery. We envisage that this framework can serve as the basis extensions to different settings including adversarial training for solving variety of problems within RL.
REFERENCES
Luluwah Al-Fagih. The british knock-out put option. International Journal of Theoretical and Applied Finance, 18(02): 1550008, 2015.
Juan Alcaraz, Jose Ayala-Romero, Javier Vales-Alonso, and Fernando Losilla-López. On- line reinforcement learning for adaptive interference coordination. Transactions Emerging Telecommunications Technologies, 31(10))44087, 2020.

---

Published as conference paper at ICLR 2023
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1-2):41-7, 2003.
Sebastian Becker, Patrick Cheridito, and Arnulf Jentzen. Deep optimal stopping. arXiv preprint arXiv:1804.00 05394, 2018.
Albert Benveniste, Michel Métivier, and Pierre Priouret. Adaptive algorithms and stochastic approxi- mations, volume 22. Springer Science & Business Media, 2012.
Dimitri P Bertsekas. Approximate dynamic programming. Athena scientific Belmont, 2012.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXtv:1606.01555 01540, 2016.
Andrew S Caplin and Daniel F Spulber. Menu costs and the neutrality of money. The Quarterly Journal of Economics, 102(4):703-725, 1987.
Shuhang Chen, Adithya M Devraj, Ana Bušić, and Sean Meyn. Zap q-learnin for' optimal stopping. In 2020 American Control Conference (ACC), pp. 3920-3925. IEEE, 2020.
Mark HA Davis and Andrew R Norman. Portfolio selection with transaction costs. Mathematics of operations research, (5())6676-713, 1990.
Abderrahim Fathan and Erick Delage. Deep reinforcement learning for optimal stopping with application in financial engineering. arXiv preprint arXiv:2105.08877 2021.
Grandt Jr. Damage tolerant design and nondestructive inspection-keys to aircraft airworthiness. Procedia Engineering, 17:236-246, 2011.
Tuomas Haamnja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.05995.5 2018.
Tommi Jaakkola, Michael Jordan, and Satinder P Singh. Convergence of stochastic iterative dynamic programming algorithms. In Advances in neural information processing systems, pp. 703 710, 1994.
Yuu Jinnai, Jee Won Park, David Abel, and George Konidaris. Discovering options for exploration by minimizing cover time. In International Conference on Machine Learning, pp. 3130-3139. PMLR, 2019.
Martin Klissarov and Doina Precup. Flexible option learning. Advances in Neural Information Processing Systems, 34:4632-4666 4646, 2021.
Ralf Korn. Some applications of impulse control in mathematical finance. Mathematical Methods of Operations Research, 50(3):493-518, 1999.
Edouard Leurent. An environment for autonomous driving decision-making https://github com/eleurent http///eureee.hiivaaaeeeeere 2018
David Mguni. Duopoly investment problems with minimally bounded adjustment costs. arXiv preprint ArXiv:1805 1974, 2018a.
David Mguni. Optimal capital injections with the risk of ruin: A stochastic differential game of impulse control and stopping approach. arXiv preprint 1005078805.01578, 2018b.
David Mguni. Optimal selection transaction costs a dynamic principal-agent problem. arXiv preprint arXiv: 01062,, 2018c.
David Mguni. viscosity approach to stochastic differential games of control and stopping involving impulsive control. arXiv preprint arXiv:1803. 7432, 2018d. 8d.
David Mguni Cutting your losses: Learning fault-tolerant control and optimal stopping under adverse risk. Xixiv preprint arXiv: 2019.
10

---

Published as conference paper at ICLR 2023
Daniel Mitchell, Haolin Feng, and Kumar Muthuraman. Impulse control of interest rates. Operations research, 62(3):602-615, 2014.
Gabricla Mundaca and Bernt Øksendal. Optimal stochastic intervention control with application to the exchange rate. Journal of Mathematical Economics, 2(29):225-243, 1998.
David C Noelle, Unsupervised methods for subgoal discovery during intrinsic motivation in model- free hierarchical reinforcement learning. In KEG@ AAAI, 2019.
Bernt Øksendal. Stochastic differential equations. In Stochastic differential equations, pp. 65 84. Springer, 2003.
Bernt Øksendal and Agnès Sulem. Approximating impulsc control by iterated optimal stopping. In Applied Stochastic Control of Jump Diffusiioss, pp. 255 272. Springer, 2019.
Jing-Cheng Pang, Tian Xu, Sheng-Y Yi Jiang, Yu-Ren Liu, and Yang Yu. Sparsity prior regularized q-learning for sparse action lasks. arXiv preprint arXiv:2105. 08566666,,, 2021.
Doina Precup, Richard S Sutton, and Satinder Singh. Theoretical results reinforcement learning with temporally abstract options. In European conference on machine learning, pp. 382-393. Springer, 1998.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1777.006344 2017.
Aivar Sootla, Diego Oyarzún, David Angeli, and Guy-Bart Stan. Shaping pulses to control bistable systems: Analysis, computation and countercxamples. Automatica, 63:254-264, 2016.
Aivar Sootla, Alexander I. Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David Mguni, Jun Wang, and Haitham Bou-Ammar. Sauté RL: Almost surely safe reinforcement learning using state augmentation. arXiv preprint 2nXiv:2202.063 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
John N Tsitsiklis and Benjamin Van Roy. Optimal stopping of markov processes: Hilbert space theory, approximation algorithms, and an application to pricing high-dimensional financial derivatives. IEEE Transactions Automatic Control, 44 (0): 1450-1851, 1999.
Pu Zhao, Yanzhi Wang, Nachyuck Chang, Qi Zhu, and Xue Lin. A deep reinforcement learning framework for optimizing fuel cconomy of hybrid electric vehicles. In 2018 23rd Asia and South Pacific design automation conference (ASP-DAC), pp. 196-202. IEEE, 2018


Reviews:
[{"review_id": "rjwd9l7ILD", "reviewer": "Reviewer_sGpp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an algorithm called LICRA to enable an RL agent to learn when to act and which actions to take while optimizing for the incurred action cost. LICRA is inspired by the method of impulse control to learn an impulse policy that decides when to act. LICRA also simultaneously learns the action selection policy that chooses the action to execute if and when the impulse policy decides to act.   \n", "summary_of_the_review": "I recommend accepting the paper if the comments above are addressed. The stated theorems in the main paper support the proposed formulation of the algorithm and the results also support the main claims made in the paper. \n", "strength_and_weaknesses": "Strengths: \n\nThe experiments are designed to clearly demonstrate the advantages of using a nested optimization approach to efficiently identify the null or no cost action that enables the RL agent to follow a pre-specified intervention or cost budget. Even when there is no cost involved, ablation results in the appendix show that LICRA is able to prioritize learning a policy for those states that ensure higher average returns. Compared to standard RL algorithms like SAC or PPO that fail to converge to an optimal policy when faced with a difficult exploration problem, LICRA consistently converges to a high reward stable solution.  \n\n\nWeaknesses:\n\n1. Sec 5, first paragraph: “..which can be evaluated online therefore allowing the g.” - This sentence seems incomplete.\n\n2. Sec 6, Fig 1: The description of Fig 1 on page 8 says “The agent can act after a time interval of 0.01 seconds and the episode ends after 75 steps.” But the x-axis in Fig 1 continues till 500k timesteps. It would help to clearly explain how to interpret the evaluation process in this setting. \n\n3. Page 8, Driving Environment Fuel Rationing : “2) if accelerations should be performed …” -> “2) accelerations should be …”?\n\n4. Fig 1 / 2 / 3 : The number of random seeds used to get the error bars is not specified.\n\n5. Fig 4: Why does the violation count increase for all cost levels when K is increased? Are the numbers in the heatmap supposed to add to 1 for each K? What is the total time of the trials?   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S7rM2CCbONP", "reviewer": "Reviewer_KqHR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on a subset of decision making problems particularly relevant to real world applications: learning _when to act_ along with the appropriate action to execute. While this problem can be framed as a standard RL problem with no-op action, the paper demonstrates why this is suboptimal. To this end, the paper proposes a new method termed LICRA that can be seamlessly applied to any existing RL algorithm to tackle such problems. The core idea is akin to hierarchical RL. However, limiting to a binary choice in switching policy simplifies many issues that make hierarchical RL difficult to make work in practice. Results are demonstrated on standard gym environments with extensions to standard deepRL algorithms.", "summary_of_the_review": "Overall even though it's overall an incremental paper, it is well executed and the problem being tackled is of high utility especially in the industry.", "strength_and_weaknesses": "**Strengths**\n\nThe paper applies ideas from hierarchical RL to help solve a class of decision making problem of wide practical importance. The idea is intuitively straightforward but well executed with applications to both on-policy and off-policy RL algorithms demonstrated on standard gym environments. Convergence and optimality guarantees are also proven in the case of application to standard Q-learning. The paper also shows how this framing allows them to do well on RL with a budget class of decision making problems.\n\n**Weaknesses**\nCalling the framework itself novel is slight overselling because it's standard hierarchical RL framing. Given the improvements in exploration with this framing, the results are not surprising.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BKHW9soSHWx", "reviewer": "Reviewer_dHaP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In many situations an RL agent incurs costs whenever it acts. Simply applying standard RL algorithms in such situations often doesn’t work, because it is not easy for the algorithm to choose “no action” instead of “small action” (particularly in continuous action spaces). To deal with this, the authors propose Learnable Impulse Control Reinforcement Algorithm (LICRA), which learns a two-step policy: the first step decides whether or not to act, and the second step decides what action to take (if any is needed). The authors provide:\n\n1. An algorithm to learn the two-step policy from experience,\n2. A proof of convergence to the optimal policy in the tabular and linear function approximator settings,\n3. An application to the setting in which there is a fixed budget of actions, and\n4. An empirical evaluation of LICRA on three environments with costly actions.", "summary_of_the_review": "While the problem setting is of interest and the algorithm proposed is simple and theoretically justified (if not particularly surprising), I would recommend rejecting, for two main reasons:\n\n1. It seems plausible that the benefits come from an inductive bias towards inaction, but this can also be achieved much more simply by changing the network architecture to add a direct prediction for “take no action” and then using standard RL algorithms. This should be a baseline to which LICRA is compared.\n2. The paper contains several incorrect claims, most worryingly claims that overstate the benefits of their method in the empirical settings they evaluate in.\n\nUPDATE: The author response has addressed many of my concerns (particularly the ones about incorrect claims). My first worry remains. As a result I am increasing my score by one step.", "strength_and_weaknesses": "Strengths:\n- The problem setting is an important one, especially when applying RL to the real world.\n- The approach in the paper is principled and elegant, while remaining fairly simple.\n\nWeaknesses:\n- It seems plausible to me that the main way in which LICRA provides a benefit is that LICRA provides an inductive bias towards inaction. It would be good to compare to simple baselines that also provide such an inductive bias: for example, use a baseline method like PPO or SAC, but extend the action space with one extra dimension corresponding to “don’t take any action”. This would test whether the extra algorithmic complexity is actually necessary.\n- The empirical evaluations of LICRA do not seem that compelling, except in the case of the Drive environment. More worryingly, the authors exaggerate the benefits of LICRA in the text (see specific examples below).\n- The authors make another incorrect claim about the asymptotic complexity of LICRA, discussed below.\n\nIncorrect claims made in the paper:\n\n> Conversely, owing to the binary decision space for g (c.f. Prop. 1), LICRA requires |S^c_I| + |S_I||A| evaluations.\n\nThis seems impossible. When you only have black-box access to a reward function R(s, a), you at least need to look at each entry in the reward function in order to get a guarantee of convergence to the optimal policy (what if the one entry you haven’t looked at yet is the highest possible reward value?), which implies that any algorithm must have complexity at least Ω(|S||A|). However this is contradicted by the claim above.\n\nPossibly the claim is about the case where you already know the set S_I? But then the size of the set S^c_I is irrelevant, since you always take the null action in that setting. (And also this is not a reasonable measure of the complexity of LICRA, because in an actual setting you don’t know the set S_I.)\n\nThis is not the only place where the authors make claims about the complexity benefits of LICRA, but it is the one where the claim is clearly incorrect. Elsewhere the claims are vague enough that they are not obviously false: for example the authors also say “By isolating the decision of whether to act or not, the LICRA framework may also reduce the computational complexity in this setting.” Nonetheless I think such statements should be removed from the paper given that they are not supported by any of the evidence presented in the paper.\n\n> The results of training are shown in Fig. 1 which clearly demonstrates that LICRA_PPO finds a better policy than standard PPO.\n\nLooking at Figure 1 it does not seem obvious that LICRA_PPO outperforms standard PPO; overall they seem to be similar. LICRA_PPO does learn a good policy faster than PPO, but this is a different claim.\n\n> Also comparing the variance among different seeds, we can see that LICRA_PPO is a much more stable algorithm than the other two.\n\n(This is also referring to Figure 1.) This claim also looks false based on Figure 1 and I am confused what the authors are seeing that I am not. As far as I can tell the blue error bars of LICRA_PPO are if anything _larger_ than the orange error bars of PPO.\n\n> In Fig. 3, we observe that the LICRA agent outperforms all the baselines, both in terms of sample efficiency and average test return (total rewards at each timestep).\n\nLooking at Figure 3, I agree that LICRA-SAC outperforms the baselines (at least on average test return), but LICRA-PPO seems comparable with SAC and PPO.\n\nMinor issues:\n\nIn the preliminaries, right after equation (1), you say:\n\n> \\mathcal{R}(s, a) = R(s, a) 1_{a∈A/{0}} + R(s, 0)(1−1_{a∈A/{0}})\n\nIsn’t this just equivalent to saying \\mathcal{R}(s, a) = R(s, a)?\n\nIn the definition of the intervention operator M, policy \\pi is associated with M and policy \\pi’ is associated with Q, but in the next sentence explaining the interpretation these are switched. Please choose a consistent usage. (Personally I find it more intuitive to have \\pi’ associated with M, since I interpret \\pi’ as the intervention policy.) I would also recommend using parentheses to clarify scopes (in particular I was initially unclear on the scope of the a_{\\tau_k} variable).\n\nJust before equation (3), you say:\n\n> Denote by M[Qπ,g] the intervention operator acting on Qπ,g when the immediate action is chosen according to an epsilon-greedy policy.\n\nShouldn’t this be a fully greedy policy, rather than epsilon-greedy, in order to satisfy the Bellman equation at optimality? (It seems like this would be needed for Theorem 1, for example.) \n\nTypo: envisge → envisage", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4jA8yDT1E8h", "reviewer": "Reviewer_DavQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a type of policy design for reinforcement learning in the presence of costs to taking actions. The authors propose to learn the decision on whether to act in a specific state separately from learning the action itself. The authors note that this setup reduces the computational complexity and therefore makes policy learning with transaction costs more effective. They further demonstrate how to adjust the method to learn policies with budget constraints by augmenting the state space and evaluate the performance of their methods in benchmark problems.", "summary_of_the_review": "I think the idea of the paper adds to the existing literature by providing a well-motivated idea on incorporating action costs into policy learning and demonstrates the effect in meaningful experiments.", "strength_and_weaknesses": "The paper has a clear underlying idea, and motivates it well both intuitively and mathematically.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster