Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In many situations an RL agent incurs costs whenever it acts. Simply applying standard RL algorithms in such situations often doesn’t work, because it is not easy for the algorithm to choose “no action” instead of “small action” (particularly in continuous action spaces). To deal with this, the authors propose Learnable Impulse Control Reinforcement Algorithm (LICRA), which learns a two-step policy: the first step decides whether or not to act, and the second step decides what action to take (if any is needed). The authors provide:

1. An algorithm to learn the two-step policy from experience,
2. A proof of convergence to the optimal policy in the tabular and linear function approximator settings,
3. An application to the setting in which there is a fixed budget of actions, and
4. An empirical evaluation of LICRA on three environments with costly actions.
strength_and_weaknesses: Strengths:
- The problem setting is an important one, especially when applying RL to the real world.
- The approach in the paper is principled and elegant, while remaining fairly simple.

Weaknesses:
- It seems plausible to me that the main way in which LICRA provides a benefit is that LICRA provides an inductive bias towards inaction. It would be good to compare to simple baselines that also provide such an inductive bias: for example, use a baseline method like PPO or SAC, but extend the action space with one extra dimension corresponding to “don’t take any action”. This would test whether the extra algorithmic complexity is actually necessary.
- The empirical evaluations of LICRA do not seem that compelling, except in the case of the Drive environment. More worryingly, the authors exaggerate the benefits of LICRA in the text (see specific examples below).
- The authors make another incorrect claim about the asymptotic complexity of LICRA, discussed below.

Incorrect claims made in the paper:

> Conversely, owing to the binary decision space for g (c.f. Prop. 1), LICRA requires |S^c_I| + |S_I||A| evaluations.

This seems impossible. When you only have black-box access to a reward function R(s, a), you at least need to look at each entry in the reward function in order to get a guarantee of convergence to the optimal policy (what if the one entry you haven’t looked at yet is the highest possible reward value?), which implies that any algorithm must have complexity at least Ω(|S||A|). However this is contradicted by the claim above.

Possibly the claim is about the case where you already know the set S_I? But then the size of the set S^c_I is irrelevant, since you always take the null action in that setting. (And also this is not a reasonable measure of the complexity of LICRA, because in an actual setting you don’t know the set S_I.)

This is not the only place where the authors make claims about the complexity benefits of LICRA, but it is the one where the claim is clearly incorrect. Elsewhere the claims are vague enough that they are not obviously false: for example the authors also say “By isolating the decision of whether to act or not, the LICRA framework may also reduce the computational complexity in this setting.” Nonetheless I think such statements should be removed from the paper given that they are not supported by any of the evidence presented in the paper.

> The results of training are shown in Fig. 1 which clearly demonstrates that LICRA_PPO finds a better policy than standard PPO.

Looking at Figure 1 it does not seem obvious that LICRA_PPO outperforms standard PPO; overall they seem to be similar. LICRA_PPO does learn a good policy faster than PPO, but this is a different claim.

> Also comparing the variance among different seeds, we can see that LICRA_PPO is a much more stable algorithm than the other two.

(This is also referring to Figure 1.) This claim also looks false based on Figure 1 and I am confused what the authors are seeing that I am not. As far as I can tell the blue error bars of LICRA_PPO are if anything _larger_ than the orange error bars of PPO.

> In Fig. 3, we observe that the LICRA agent outperforms all the baselines, both in terms of sample efficiency and average test return (total rewards at each timestep).

Looking at Figure 3, I agree that LICRA-SAC outperforms the baselines (at least on average test return), but LICRA-PPO seems comparable with SAC and PPO.

Minor issues:

In the preliminaries, right after equation (1), you say:

> \mathcal{R}(s, a) = R(s, a) 1_{a∈A/{0}} + R(s, 0)(1−1_{a∈A/{0}})

Isn’t this just equivalent to saying \mathcal{R}(s, a) = R(s, a)?

In the definition of the intervention operator M, policy \pi is associated with M and policy \pi’ is associated with Q, but in the next sentence explaining the interpretation these are switched. Please choose a consistent usage. (Personally I find it more intuitive to have \pi’ associated with M, since I interpret \pi’ as the intervention policy.) I would also recommend using parentheses to clarify scopes (in particular I was initially unclear on the scope of the a_{\tau_k} variable).

Just before equation (3), you say:

> Denote by M[Qπ,g] the intervention operator acting on Qπ,g when the immediate action is chosen according to an epsilon-greedy policy.

Shouldn’t this be a fully greedy policy, rather than epsilon-greedy, in order to satisfy the Bellman equation at optimality? (It seems like this would be needed for Theorem 1, for example.) 

Typo: envisge → envisage
clarity,_quality,_novelty_and_reproducibility: Quality: As mentioned in the weaknesses above, there are many correctness issues with this paper. In addition I would like to see a comparison to simple baselines that incorporate an inductive bias towards inaction (also discussed in the weaknesses).

Clarity: Overall I found the paper to be well explained and clear (except in the cases where incorrect claims were made which of course led to confusion).

Originality / Novelty: I do not know of other work tackling this setting, and if the approach did significantly outperform a good baseline, I think that would be a significant contribution.
summary_of_the_review: While the problem setting is of interest and the algorithm proposed is simple and theoretically justified (if not particularly surprising), I would recommend rejecting, for two main reasons:

1. It seems plausible that the benefits come from an inductive bias towards inaction, but this can also be achieved much more simply by changing the network architecture to add a direct prediction for “take no action” and then using standard RL algorithms. This should be a baseline to which LICRA is compared.
2. The paper contains several incorrect claims, most worryingly claims that overstate the benefits of their method in the empirical settings they evaluate in.

UPDATE: The author response has addressed many of my concerns (particularly the ones about incorrect claims). My first worry remains. As a result I am increasing my score by one step.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an algorithm called LICRA to enable an RL agent to learn when to act and which actions to take while optimizing for the incurred action cost. LICRA is inspired by the method of impulse control to learn an impulse policy that decides when to act. LICRA also simultaneously learns the action selection policy that chooses the action to execute if and when the impulse policy decides to act.   

strength_and_weaknesses: Strengths: 

The experiments are designed to clearly demonstrate the advantages of using a nested optimization approach to efficiently identify the null or no cost action that enables the RL agent to follow a pre-specified intervention or cost budget. Even when there is no cost involved, ablation results in the appendix show that LICRA is able to prioritize learning a policy for those states that ensure higher average returns. Compared to standard RL algorithms like SAC or PPO that fail to converge to an optimal policy when faced with a difficult exploration problem, LICRA consistently converges to a high reward stable solution.  


Weaknesses:

1. Sec 5, first paragraph: “..which can be evaluated online therefore allowing the g.” - This sentence seems incomplete.

2. Sec 6, Fig 1: The description of Fig 1 on page 8 says “The agent can act after a time interval of 0.01 seconds and the episode ends after 75 steps.” But the x-axis in Fig 1 continues till 500k timesteps. It would help to clearly explain how to interpret the evaluation process in this setting. 

3. Page 8, Driving Environment Fuel Rationing : “2) if accelerations should be performed …” -> “2) accelerations should be …”?

4. Fig 1 / 2 / 3 : The number of random seeds used to get the error bars is not specified.

5. Fig 4: Why does the violation count increase for all cost levels when K is increased? Are the numbers in the heatmap supposed to add to 1 for each K? What is the total time of the trials?   

clarity,_quality,_novelty_and_reproducibility: The proposed algorithm is simple but clearly shows the benefits of using an impulse policy to learn when to act along with simultaneous learning of a separate policy that decides the action to take. The paper is easy to read and the writing is overall easy to understand, barring the points I have mentioned above in the weaknesses. 

summary_of_the_review: I recommend accepting the paper if the comments above are addressed. The stated theorems in the main paper support the proposed formulation of the algorithm and the results also support the main claims made in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a type of policy design for reinforcement learning in the presence of costs to taking actions. The authors propose to learn the decision on whether to act in a specific state separately from learning the action itself. The authors note that this setup reduces the computational complexity and therefore makes policy learning with transaction costs more effective. They further demonstrate how to adjust the method to learn policies with budget constraints by augmenting the state space and evaluate the performance of their methods in benchmark problems.
strength_and_weaknesses: The paper has a clear underlying idea, and motivates it well both intuitively and mathematically.
clarity,_quality,_novelty_and_reproducibility: While the idea of impulse control and balancing costs with optimal actions is no particularly novel idea, the authors proposal to incorporate it into RL methods appears original and is well motivated and clearly laid out.
summary_of_the_review: I think the idea of the paper adds to the existing literature by providing a well-motivated idea on incorporating action costs into policy learning and demonstrates the effect in meaningful experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on a subset of decision making problems particularly relevant to real world applications: learning _when to act_ along with the appropriate action to execute. While this problem can be framed as a standard RL problem with no-op action, the paper demonstrates why this is suboptimal. To this end, the paper proposes a new method termed LICRA that can be seamlessly applied to any existing RL algorithm to tackle such problems. The core idea is akin to hierarchical RL. However, limiting to a binary choice in switching policy simplifies many issues that make hierarchical RL difficult to make work in practice. Results are demonstrated on standard gym environments with extensions to standard deepRL algorithms.
strength_and_weaknesses: **Strengths**

The paper applies ideas from hierarchical RL to help solve a class of decision making problem of wide practical importance. The idea is intuitively straightforward but well executed with applications to both on-policy and off-policy RL algorithms demonstrated on standard gym environments. Convergence and optimality guarantees are also proven in the case of application to standard Q-learning. The paper also shows how this framing allows them to do well on RL with a budget class of decision making problems.

**Weaknesses**
Calling the framework itself novel is slight overselling because it's standard hierarchical RL framing. Given the improvements in exploration with this framing, the results are not surprising.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and well executed. While not novel per se, the approach would be of wide practical importance and demonstrating _how_ it works with existing standard RL algorithm is super useful. However unclear if the source code of the experiments will be released.
summary_of_the_review: Overall even though it's overall an incremental paper, it is well executed and the problem being tackled is of high utility especially in the industry.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Below are the pros and cons of the paper, which the reviewers in the discussion meeting all agreed on. This paper is generally borderline and no reviewer had strong opinions about acceptance or rejection,

Pros:
- The method is a clever idea that is generally well-done and well explained.
- The experiments illustrate the benefit of the key idea.

Cons:
- In continuous action settings, it's not clear if the success of the method is simply because "it is far easier for it to utilise the null action of zero acceleration/braking than the other algorithms". A simple baseline of including a discrete "null" action would help provide more insight to where the gains are coming from. Though, it is good that the authors seem to be open to adding ablations to this experiment, and the discrete action experiments do show improvements.
- At least one reviewer suspects that they would not use this method if they encountered the problem studied in this paper and instead opt for the simple baseline above in continuous action settings. Though, experiments on the setting might alleviate that concern if the method showed strong improvements.
- The experimental problem domains are all rather simplistic/toy. It's not clear if experiments on three simple domains, one of which does not show a particularly convincing improvement, meets the bar for ICLR. Experiments on larger scale and/or real problems (and also just more problem domains) would make the impact of the method even more convincing.

As a side note, the lunar lander results improvements look quite small at first glance, but when looking more closely at the y-axis, the reviewers & I realized that the improvements are larger than they first appear. I might recommend to zooming in on the y-axis, e.g. starting it at -75, so that it is easier to see the differences.
justification_for_why_not_higher_score: - In continuous action settings, it's not clear if the success of the method is simply because "it is far easier for it to utilise the null action of zero acceleration/braking than the other algorithms". A simple baseline of including a discrete "null" action would help provide more insight to where the gains are coming from. Though, it is good that the authors seem to be open to adding ablations to this experiment, and the discrete action experiments do show improvements.
- The experimental problem domains are all rather simplistic/toy. It's not clear if experiments on three simple domains, one of which does not show a particularly convincing improvement, meets the bar for ICLR. Experiments on larger scale and/or real problems (and also just more problem domains) would make the impact of the method even more convincing.

justification_for_why_not_lower_score: - The method is a clever idea that is generally well-done and well explained.
- The experiments illustrate the benefit of the key idea.
Decision: Accept: poster