Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper deals with the question namely the issue when an optimal
continuous ("soft") control can be superseded/replaced or even needs
to be implemented by a Bang-Bang control.
main_review: The paper deals with an interesting question, namely the nature of
optimal control solutions and whether a simplistic bang-bang might
work just as well as a continuous (e.g. Gaussian) control. The reviewer
particularly likes the idea to ask what the nature of the problems
would be which would favour bang-bang, rather than just presenting
"yet another algorithm" - in this sense, this paper is researching the
*nature* of problems leading to particular optimal effects. The paper
also is careful to note that RL entangles exploration, learning and
execution, and the issue of trade-offs. Amongst other, action costs
may discourage bang-bang, but for Gaussian exploration they may
confine the exploration; it is clear that there are many subtleties
that dictate the preferred outcome.

A very compact derivation (based on Pontryagin) and discussion of
emergence of bang-bang and a set of examples where bang-bang is
predominantly seen in a number of "walking"-type scenarios.

A number of instructive analyses demonstrate the prevalence of various
solutions and dynamics under varying action costs and thus balances
and thus sheds light on the nature of problems and cost structures
that favour bang-bang against soft controls. An interesting research
paper on the meta-question of what solutions one can expect for
certain problem formulations.

I have no particular comments on this well-written paper.

limitations_and_societal_impact: Limitations etc. are very well discussed throughout.

As for Societal Impact, this is essentially basic research; there should be an N/A option on the "Societal Impact" statement. Asking authors of basic research to comment on societal impact, unless it is directly relevant is no better than asking them to read entrails.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors analyze the emergence of implicit bang-bang (action saturation) control behavior in Deep RL. They claim three contributions, i) they argue that these are often actually optimal by making a connection to time-optimal solutions in optimal control theory, ii) they show empirically that the performance of an inherently bang-bang (discrete action) control policy yields comparable performance, and iii) they discuss possible extensions to reduce bang-bang behavior and their connection to exploration . 

main_review: The emergence of undesired bang-bang behavior can be a problem in real-world applications since it can damage mechanical systems or cause instability. Any insight into this problem would be valuable for control applications. Clarity of presentation is adequate and the paper contains a wealth of empirical experiments which appear to be carefully documented.

However, I have some concerns about the quality of the three main contributions as outlined below:
1) The first contribution is a proof from optimal control theory that bang-bang is optimal in continuous-time minimum-state problems without action penalties. The theoretical contribution here seems rather minor since it appears to follow directly from a book on optimal control [23]. Anybody that has played around with MPC also won't be surprised that the controller will saturate any action limits if the action penalty is low enough.  
However, shouldn't pure bang-bang technically only be *optimal* for continuous-time, where max control output can be countered by an arbitrary small time step? For discrete-time, if the state is close enough to the target, the optimal action could be anywhere in-between the action limits. The authors claim that this continuous-time behavior translates to discrete-time systems because the time steps in RL are generally small enough, but this is never properly motivated. Even without examining this, running at higher Hz could of course be a solution, but it requires more compute and that the state is also available at that rate.  
 The paper does raise a valid point about the importance of including action penalties in benchmark design, otherwise the agent will certainly saturate the action constraints, which may be undesirable in practice. Although the authors use the DeepMind control benchmarks as examples here, to the best of my knowledge, all the gym Mujoco benchmarks actually do include some kind of action penalty. I don't know if they are suitable or not. It also seems unlikely that bang-bang behavior observed in applications is just because they forgot to include an action penalty. This avenue is never really explored, and the paper never precisely defines what type of bang-bang behavior they are trying to fix - is it all kinds of action saturation, or just quick oscillations near an equilibrium? The latter may sometimes actually be desirable, but the latter never is.

2) The second contribution is empirical experiments showing that a discrete-action (bang-bang) control policy achieves performance comparable to regular continuous Gaussian ones. However, this just proves that current methods with Gaussian policies perform as poorly as a bang-bang controller. If the Gaussian ones often exhibit bang-bang behavior as purported in the motivation to this paper, this result does not appear particularly surprising. Finally, this focuses on training curves, which is not the same thing as the final performance. The tracking error near an equlibrium will often be dominated by the penalties to get there, especially if a square norm is used. It is unfortunate that they never mathematically define what constitutes harmful bang-bang behavior. The only attempt at measuring this are the action distribution plots in Figure 2, but it is difficult to see what is spurious oscillation and action saturation due to design of the reward function (low action penalty). 

3) The third contribution is a purported discussion of possible extension to reduce bang-bang behavior by including action penalties. I didn't find anything supporting this except the recommendation to introduce action penalties. An important observation, but I suspect many practitioners with control background were already aware of this.  
The authors also argue that action penalties can impede exploration and try to disentangle these. I did not find this line of reasoning entirely convincing. Couldn't changing the action penalties also drastically change the optimal strategy (e.g. run vs. walk cycles are quite different), so even if the lack of action penalties frees it up to visit more of the state space, it may be learning in the wrong region of policy space (e.g. you really wanted an energy efficient walk, not a run)? Are plots comparing state coverage meaningful here?

In conclusion, the authors target an important problem but the contributions appear minor or vague. The main finding only seems to pertain to the benchmarks in the DeepMind control suite that lack action penalties. The paper fails to convince me that a simple lack of action penalties is the main reason why DeepRL policies sometimes exhibit harmful oscillations, in the real world or even in other common control benchmarks.

Some details:

L18-54 - Intro and motivation is a bit fuzzy with several claimed minor contributions

L91: "The continuous time-setting simplifies analysis and provides a good approximation under the high sampling rates that are common on continuous control benchmarks and real-world robotic system." - One would rather see the discrete time as an approximation to continuous time when it comes to control tasks.

L112: "Nevertheless, we find that in this work the discretizaton intervals are small enough to not affect optimality of the learned policies." - based on what?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The work investigates emergence of bang-bang control policies in
continuous RL problems, by making the connection to optimal control.
Exmpirically, it shows how continuous control problems are solved with
bang-bang policies, using 4 different algorithms, and that for most of
the domains in these experiments, a continuous action space is not
needed.
The experiments also explore modified reward functions designed to
increase smootheness of the learnt policies and their effect on the
(original) return obtained during exploitation.

main_review: Originality: the main contribution of the work is not a new method but
rather a new investigation of existing techniques about emergence of
bang-bang policies, with the goal to provide insights into designing
RL experiments and rewards. Previous work (that is also referenced)
focuses on approaches helping to avoid such policies but there is no
previous investigation of the kind presented in this work.

Quality: To my understanding the work is technicaly sound and supports
the claims, though is missing some details that would be good to
see. It consists of a theoretical analysis that provides an
explanation for emergence of bang-bang control, though without	
complete proofs in the main paper. The experimental part would benefit
from more details (though parameters and some other details are
provided in the supplementary material): I may have missed it but
didn't see how often each experiment was run (or if each was only run
once). The work	presents a complete and carefully (and as far as I can
judge honestly) considered investigation. It does leave	open questions
for how to design rewards, benchmarks, or for how to avoid bang-bang
policies, and does not present a	solution to all	these, but rather
provides a basis for future work in these directions.

Clarity: The paper is clearly written and very well organised, and
presentation including graphs is of high quality.

Significance: I do believe the results are important, and the work is
set to be useful for future work in a number of directions - e.g., for
learning controllers where bang-bang policies a	better avoided like in
robotics, or potentially also for approaches speeding up
exploration. The work is not yet another paper improving
state-of-the-art on some benchmark (even though SOTA on some
benchmarks is achieved with the Bernoulli distributions used in the
experiments) but providing some new insights that I find more valuable
because of wider implications.

limitations_and_societal_impact: Limitations and societal impact are adequately addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper focuses on the so-called bang-bang policies. These are ones, which are allowed only for discrete values, typically two extremal ones. Even though they are usually disadvantageous in real-world scenarios, they tend to arise in simulation, and they seem to deliver at least decent performance.
main_review: The paper focuses on the so-called bang-bang policies. These are ones, which are allowed only for discrete values, typically two extremal ones. Even though they are usually disadvantageous in real-world scenarios, they tend to arise in simulation, and they seem to deliver at least decent performance. I find the paper overall interesting and valuable; it is all generally well written. It presents a theoretical analysis and proper empirical evaluation. Below I list the pros and cons and some (typically minor) recommendations and questions. 



Pros:

* Choices of algorithms covers well the spectrum of the current state-of-the-art
* The authors attempt to make a deeper analysis than just observing the basic phenomenon
* Related work seems fairly complete (I am not an expert though, see also the cons section)
* The paper is overall well-structured, well-written and easy to follow (with exceptions mentioned in the cons section)
* In general, I appreciate additional experiments 



Cons:

* The papers fall into a benchmark-like category. As such, there are higher expectations as to the empirical section. I realise that it is always a hard choice, and one needs to stop. Still, I'd appreciate having more environments. Moreover, the authors use $4$ seeds, which might be too small for RL settings. 
* I find Figure 1 somewhat overwhelming. I appreciate putting all the training data. However, I'd find a useful summary. 
* I am not quite convinced about the exploration section. It is almost by design that bang-bang spans a bigger region (which the authors mention). If the conclusion is that this is an issue for exploration, I'd appreciate algorithmic developments towards increasing exploration in the Gaussian setting.
* Similar problems, though from a different perspective, were studied in https://arxiv.org/pdf/1910.02208.pdf. I'd suggest the authors should at least mention this in Related Word or, better, compare SOP with bang-bang SOP.
* I find Action Penalty somewhat incomplete and incomprehensive (again, I am overwhelmed by Fig 7. )
* (Nitpick) The authors do not specify technical details of the environments. It'd be nice to make the paper self contain. 
* (Nitpick) I'd appreciate more details about discretised SAC than Alg 2 (say what is bijector)
* (Nitpick) It is perhaps good to mention that discretisation was superior in a popular Learning to run contest: "https://arxiv.org/abs/1804.00361"



Questions and suggestions:

* I acknowledge the theoretical section. I'd appreciate some discussion on how much the studied model is relevant to the environments in the paper and real-world scenarios. 
* (Nitpick) I'd appreciate having more hypotheses, why bang-bang work so well.



Originality: The paper is mostly a benchmark paper. 

Quality: Yes, this is a fair empirical work with some theoretical background.

Clarity: The paper is clear.

Significance: Comprehensive study of the bang-bang policies presented in the submission can be beneficial for the community. 
limitations_and_societal_impact: I am not satisfied with the limitation description. Generally, I am fond of merging this with 'the further research discussion' (as the authors did). However, in this case, it is blurred to the point, I do not know what the authors think about the limitations of this paper.
review_title: null

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The authors' answers have been appreciated by the reviewer, who reached a consensus about accepting this paper.
I want to congratulate the authors and invite them to modify their paper following the reviewers' suggestions.
Decision: Accept (Poster)