Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: According to the authors, this paper studies the problem of choosing when to act, also known as the impulse control problem. Choosing to abstain from actions has relevance in various real-world scenarios where doing nothing can be the right decision. As the authors discuss, this can be because of some costs associated with taking action. A motivating example of their work is portfolio management, where not buying or selling financial instruments is the right move as fees are associated with each action.

The authors model propose a framework that incorporates the impulse control problem in the reinforcement learning problem. They accomplish this by representing costs associated with actions taken in the reward function. They decompose the agent policy into two policies: the original action space policy and a binary policy to choose whether to act or not. They argue against including this action in the rest of the action set because it would be inefficient in the learning process due to action cardinality. 

The authors further prove theoretical results that their approach will converge to the optimal Q function. They do this in the tabular and linear function approximator cases. Their experiments further validate the framework, and the authors suggest their framework appropriately deals with these challenges empirically compared to baselines. 

strengths_and_weaknesses: According to the authors, the paper presents a significant problem that has not been studied in the reinforcement learning problem. From the reviewer's experience in robotics, it is a valid consideration that not sending a new action can be the optimal decision. The author's theoretical results support the validity of their proposed algorithms. The fact that their algorithm can augment existing reinforcement learning algorithms is a great advantage as it means one can quickly implement the additional policy for real-world applications. The way the authors structured their problems seems appropriate to understand the problem they are interested in addressing. 


However, a concern with this paper is with the experiment evaluations. Much more evaluation is necessary to validate the LICRA framework. A few crucial things that come to mind would be to validate the theoretical results in the tabular and linear cases. The authors discuss Q-learning-like algorithms, but it seems like they do not even propose it by itself as a new algorithm or do any experiments with it. Not evaluating this update rule feels like a missed opportunity to isolate and understand the influence of the LICRA framework in more easily reproducible settings.

Supposing the there were reasonable justification against these types of experiments, their existing experiments could still be improved. One discrepancy in the experiments is the different versions of LICRA algorithms that the authors claim to study instead of report. In the Experiments introduction, they introduce LICRA_PPO and LICRA_SAC, but in Figure 1, LICRA_SAC is not present; in Figure 2, there is just "LICRA,"; and only in Figure 3 are both of these algorithms present. Perhaps this is nit-picking, but from the author's inclusion of different combinations of RL policies with their algorithm, one would expect full coverage of the combinations (so four different versions of LICRA if using both SAC and PPO for each sub-policy). Additional comments are left for the question section as they are more about details to be included and for clarification. 
questions: - Are features shared for both policies in the experiments or are they entirely separate?
- Can the action decision policy be a separate linear head trained with a different reward signal?
-  Were any experiments done to validate the theoretical results?
- A statement in section 4 seems important to evaluate: "if the ratio of s to s_i is large, learning g before pi is essential". What was the rationale for abstaining from testing this scenario? 
- In theorem 2, the max is between taking action or future expected reward after doing nothing. In terms of having taken action, why is it a_t and not max_{a_t}? Wouldnt this make this make it more of a SARSA update?
- What is delta in section 6 Equation 4? I could not find the definition.
- What is the state variable of the Merton problem? Does it have any sequential relations?
- In the Merton experiments, are the results sensitive to the environment parameters of the SDE? I.e., is LICRA robust to these choices?
- Which hyperparameters were tuned over? It is mentioned in the 12 Hyperparameter section that "square brackets indicate ranges," but I did not see any in the hyperparameter in the table using square brackets.
- Equation 14 of proof of theorem 1, inconsistent usage of lambda (is its function of 1 or 2 inputs?). What does "I(\tau)" mean in Equation 14?
- Please tighten up notation in theorem one proof; the proof starts with \lambda, which becomes \phi and then \theta. It is not unclear why these variables change.
- A paper that might be related is the following: Nonparametric learning for impulse control problems Soren Christensen and Claudia Straugh. Disclaimer: I thoroughly read the paper, but they mention impulse control and reinforcement learning 
- Would it be possible to include in the preliminary discussion on Impulse Control Problem? This seems central to your research but is omitted in the paper.
- Consider restructuring your existing experiments section. It feels like Figures 1, 2, and 3 are, in some sense, experiments of similar conclusions (i.e., our method performs well). The ablations you do in the appendix seem valuable to include in the main paper as it isolates the effects of hyperparameters of the framework.
- Consider moving the less crucial details of the Merton problem to the appendix. For example, the SDE feels unnecessary in the main paper. 
- You reference Fig. 5 in line 462 when it seems you mean Figure 4. 
- Consider removing the examples in "3 Preliminaries". They do not add much to the discussion that was not already mentioned in the introduction. 
- Consider using a more digestible metric for performance over learning curves (e.g. Final average performance). It’s a bit difficult to distinguish the performance gains of your algorithm.
- Include Compute resources used. This is mentioned as required in the Checklist, but these details weren’t mentioned in the appendix or paper.
- Figure 6: If you change to a single metric (average final reward, the area under the curve, success rate, etc.) You could consider using a radar plot to visualize performance. As higher is assumed to be better, you could have the rewards on the outside, and each radar is your algorithm. The bigger the circle, the better your algorithm is across cost functions.
- What is Figure 5 showing exactly? As K goes up, fewer violations should occur as it's terrible to decelerate in these zones, but it wasn't clear. It could help to visualize this experiment to explain it. 

limitations: It's hard to gauge the limitations of their work because the authors do not seem to discuss this explicitly in the paper. In terms of negative societal impact, there doesn't seem to be any apparent reason their work could have negative social impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an algorithm for finding the optimal policy in sequential decision problems where there is a cost associated with taking an action. At each timestep, the agent can choose to perform an action or do nothing. LICRA learns two RL policies: one for deciding when to act, and another for taking actions. By first learning when to act, the complexity of the problem is reduced compared to standard RL configurations. 
strengths_and_weaknesses: #### Originality
The authors present a novel method for impulse control using two RL policies.

#### Quality
The paper does a nice job of showcasing practical applications of LICRA in environments such as finance and autonomous driving. I did not examine the proofs in the appendix very closely, but the results seem to follow. 

I think the related work section is lacking descriptions and citations for related problem settings such as RL safety and hierarchical RL. Many of these works use a safety policy or high-level policy to intervene the lower-level policy during training. Since LICRA trains two RL processes, it would be interesting to see how it compares to other methods that do the same.

#### Clarity
The paper is well structured and easy to understand. Algorithm 1 presents a procedure that should be easy to implement with standard auto-differentiation libraries. During the exposition of LICRA, the authors state that adding the zero action is not a good solution to impulse control. I think a small motivational example could be good for demonstrating this. 

##### Nits
* The last paragraph of the introduction should be rewritten for grammar and clarity
* Line 28: result "in" catastrophic losses
* Line 56: series of results
* Line 82: form of
* Line 90: task is to
* Line 135: not significantly impact
* Line 190: $P(\cdot ;  a_{\tau_k}, s_{\tau_k})$

#### Significance
The results are useful for applications involving impulse control. Researchers or practitioners may use LICRA due to its effectiveness and how easy it is to implement in practice.
questions: * In Figure 1, what are the measures of centrality and spread for the different algorithms? How many random seeds were used? How much hyperparameter tuning was necessary to achieve the results shown. Same questions for Figures 2 and 3.
* Please provide brief descriptions of each baseline algorithm used, and why they were chosen to be compared with LICRA.
* What do minibatches refer to in Appendix 12?
* What do optimization epochs refer to in Appendix 12?
* In Appendix 12, there are no hyperparameters in square brackets following the notice in the second sentence. This section seems very incomplete to me. The SAC and PPO variants of LICRA should have different sets of hyperparameters. Are these hyperparameters also used for the baseline algorithms? Are these hyperparameters the same for both $\pi$ and $g$?
* Are the costs included in the Average Episodic Return reported in the figures? It could be insightful to compare the baselines using the vanilla reward.
limitations: I think the potential limitations of LICRA should be discussed in more detail. The experimental environments chosen are very well suited to LICRA since they involve the agent positioning itself in a certain configuration (e.g., spaceship over the goal in lunar lander) and maintaining that configuration by not acting. It would be nice to see how LICRA performs in environments that are less suited to it. For example, what about tasks where the agent should be acting “most” of the time, making it harder for $g$ to learn the few states where it should intervene?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies a class of sequential decision making problems, in which taking an action can incur a large cost. To address these problems, they proposed a method named Learnable Impulse Control Reinforcement Algorithm (LICRA), which learns both when to act and which action to take. For the proposed algorithm, they introduced a new Bellman operator and established convergence results. Then they generalized LICRA with state augmentation to solve the RL problem with a budget. Finally, they tested LICRL on three tasks.  
strengths_and_weaknesses: Strengths:
1. They proposed a reasonable LICRA algorithm that utilizes two separate policies to learn when to act and which action to take. It was shown that LICRA can also solve the RL problem with a budget.


Weakness:
1. The proposed method LICRA is not well-motivated and it is unclear why we need to learn two separate policies. As mentioned by the authors, to address RL problems in which taking an action can incur a cost, we can simply augment the action space with a null action and apply classical RL algorithms. Compared with this simple method, LICRA has an identical search space (action space) but requires twice the computational cost for learning the additional actor and critic. Besides, constrained RL approaches can also be applied in this setting. The authors do not show the advantages of LICRA over the methods mentioned before.
2. The empirical evaluations are weak. First, the algorithms for comparison are too few. The constrained RL algorithms should be involved for comparison. Second, the improvements of LICRA are not clear on Merton’s Portfolio Problem and Lunar Lander task. Third, the introductions of the baseline algorithms in Figure 2 and Figure 3 are missing.
3. The theoretical results and notations in Sec 5 are misleading. See the question part for details. I encourage the authors to re-write this part to improve the clarity.  
4. This paper studies RL problems where taking an action can incur a large cost. A closely related work [1] is missing. [1] studies the RL problems where the amounts of some specific actions are limited, which is similar to the problems studied in this paper.

References:
[1]. Ji-Cheng, Pang, et al. "Sparsity Prior Regularized Q-learning for Sparse Action Tasks.”arXiv preprint, arXiv: 2105.08666, (2021).

questions: 1. The operator of $\mathcal{M}^{\pi, g}$. This operator should be applied on **any** function in $S \times A \rightarrow \mathbb{R}$. Why use the notation $Q^{\pi, g}$, which depends on $(\pi, g)$? Besides, the function $\mathcal{M}^{\pi, g} Q^{\pi, g}$ should be dependent on $Q^{\pi, g}$. However, according to the definition given in line 186, $\mathcal{M}^{\pi, g} Q^{\pi, g}$ is independent of $Q^{\pi, g}$. Besides, in line 191, what does the notation $\mathcal{M} v^{\pi, g}$ mean?
2. The operator T in line 195. The operator T does not depend on any policy $(\pi, g)$. However, in its definition, the terms $\mathcal{M}^{\pi, g}$, $Q^{\pi, g}$ and $v^{\pi, g}$ all depend on some policy $(\pi, g)$, 

limitations: The authors do not mention any limitations or potential negative societal impact of their work.
review_title: null

Meta-review: metareview: After careful consideration, I feel I must reject this paper.  This paper essentially proposes a specific hierarchical structure onto the action space for a specific set of MDPs (which motivated the paper), where all actions apart from one (the 'do nothing' action) incur some cost.

The proposed mechanism is a reasonable solution technique for such problems.  But the fact that hierarchical methods for specific problems that inject a form of domain knowledge are somewhat better than methods that do not has been shown many times, and is unsurprising.

The proposed method is, for instance, brittle to slight variations of the problem: what if some actions sometimes do or do not incur costs, for instance, rather than there being one clear such action? If you do nothing to a robot and it then falls off the table, does or does that not count as an immediate cost? Also, the framing of the paper excludes the 'do nothing' action from the 'original' action space, which - as discussed with reviewers - is a strange framing: why was a valid action, which apparently is often the optimal choice, excluded from the original action space?  And, more importantly, what if we just add it back?

I found the explanations of the authors for why to use the proposed mechanism rather than considering the flat action space to be ultimately unconvincing and, more importantly, rather domain specific and brittle.  One could argue that the proposed mechanism is sometimes a useful inductive bias, for some problems, which is not very surprising.  Other action decompositions will be useful for other problems.

Ultimately, I think the paper should 1) make a better case for why and when this particular hierarchy is important and beneficial compared to just allowing the system to decide for itself when to take each action (including the 'do nothing' action), 2) should be more upfront about limitations (e.g., when is this particular inductive bias a bad idea, rather than beneficial?), and 3) position itself better in existing work (e.g., show more awareness about prior hierarchical approaches - I would argue this is not really a 'novel method', but instead a specific instance of a hierarchical RL system).

I will therefore suggest to reject the paper so that the authors have an opportunity to use these comments to improve the paper.
confidence: Less certain
Decision: Reject