Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies
Authors: Tim Seyde
Igor Gilitschenski
Wilko Schwarting
Bartolomeo Stellato
Martin Riedmiller
Markus Wulfmeier
Daniela Rus
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
optimal control
robotics

Abstract: Reinforcement learning (RL) for continuous control typically employs distributions whose support covers the entire action space. In this work, we investigate the colloquially known phenomenon that trained agents often prefer actions at the boundaries of that space. We draw theoretical connections to the emergence of bang-bang behavior in optimal control, and provide extensive empirical evaluation across a variety of recent RL algorithms. We replace the normal Gaussian by a Bernoulli distribution that solely considers the extremes along each action dimension - a bang-bang controller. Surprisingly, this achieves state-of-the-art performance on several continuous control benchmarks - in contrast to robotic hardware, where energy and maintenance cost affect controller choices. Since exploration, learning, and the final solution are entangled in RL, we provide additional imitation learning experiments to reduce the impact of exploration on our analysis. Finally, we show that our observations generalize to environments that aim to model real-world challenges and evaluate factors to mitigate the emergence of bang-bang solutions. Our findings emphasise challenges for benchmarking continuous control algorithms, particularly in light of potential real-world applications.

Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies
Tim Seyde' MIT CSAIL
Igor Gilitschenski University of Toronto
Wilko Schwarting MIT CSAIL
Bartolomeo Stellato Princeton University
Martin Riedmiller
Markus 2 DeepMind
Daniela Rus MIT CSAIL
DeepMind
Abstract
Reinforcement learning (RL) for continuous control typically employs distributions whose support covers the entire action space. In this work, we investigate the colloquially known phenomenon that trained agents often prefer actions the boundaries of that space. We draw theoretical connections to the emergence of bang- bang behavior in optimal control, and provide extensive empirical evaluation across variety of recent RL algorithms. We replace the normal Gaussian by Bernoulli distribution that solely considers the extremes along each action dimension a bang-bang controller. Surprisingly, this achieves nfte-of-the-art performance several continuous control benchmarks in contrast to robotic hardware, where energy and maintenance cost affect controller choices. Since exploration, learning, and the final solution are entangled in RL, we provide additional imitation learning experiments reduce the impact of exploration on our analysis. Finally, we show that our observations generalize to environments that aim to model real- world challenges and evaluate factors mitigate the emergence of bang-hang solutions. Our findings emphasise challenges for benchmarking continuous control algorithms, particularly in light potential real-world applications.
Introduction
Real-world robotics tasks commonly manifest as control problems over continuous action spaces. When learning to act in such settings, control policies are typically represented as continuous probability distributions that cover all feasible control inputs often Gaussians. The underlying assumption is that this enables more refined decisions compared to crude policy choices such as discretized controllers, which limit the search space but induce abrupt changes. While switching controls can be undesirable in practice they may challenge stability and accelerate system wear- down, they are theoretically feasible and even arise as optimal strategies in some settings. It is therefore important to investigate our underlying assumption in designing policies for learning agents, and analyze how deviations from expected behavior can be explained.
Practitioners have empirically observed that even under Gaussian policies extremal switching, or bang-bang control [31], may naturally emerge (e-g. in 51,)).. Thus, recent work focused on developing methods for preventing this behavior [7] improving training when bang-bang control is the optimal policy structure [28]. However, understanding the performance, extent, and reasons for emergence of Bang-Bang policies in RL is largely open research question. Its answer is important for designing future benchmarks and informing empirical and theoretical RL research directions.
'Correspondence to taeyde@mit. edu. "Equal advising. Please find videos and additional details https: www.g/......ieee/..../ wwwp.///w//waanbbaaeeeeeee
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

In this paper we address these questions in two ways. First, we provide a theoretical intuition for bang-bang behavior in reinforcement learning. This is based on drawing connections to minimum- time problems where bang-bang control is often provably optimal. Second, we perform a set of experiments which optimize controllers via on-policy and off-policy learning as well as model-free and model-based ofte-of-the-art RL methods. Therein we compare the original algorithms with a slight modification where the Gaussian policy head is replaced with the Bernoulli distribution resulting in bang-bang controller.
In addition to theoretical justifications, our empirical results confirm emergence of bang-ban policies in standard continuous control benchmarks. Across the board, our experiments also demonstrate high performance of explicitly enforced bang-bang policies: the modified policy head can even outperform the original method. In connection to optimal control theory. we demonstrate how action costs empirically lead to sub-optimal performance with bang-bang controllers. However, we also show the negative impact of action penalties on exploration with Gaussian policies. Due to the necessity of exploration in RL to find the optimal solution, this can result a complex trade-off.
We also provide a discrete action space version of Maxmimum A Posteriori Policy Optimization (MPO) [1] that avoids relaxations or high variance gradient estimators, which many algorithms rely on when replacing Gaussians with discrete distributions. This is particularly useful as exploration, learning process and final performance are highly entangled in RL, and inaccurate or biased gradients can strongly affect learning. We furthermore aim to mitigate this entanglement by including results for distilling behaviour of a trained Gaussian agent into both a continuous and a discrete agent to compare their performance and provide further evidence for emergent bang-bang behavior.
In summary, work contains the following key contributions:
1. We show competitive performance of bang-bang control on standard continuous control benchmarks by adapting several recent RL algorithms to leverage Bernoulli policies.
2. We draw theoretical connections to optimal control, motivating the emergence of bang-bang behavior in certain problem formulations even under continuous policy parameterizations.
3. We discuss the introduction of action penalties as a common method to reduce the emergence bang-bang behaviour, and highlight resulting trade-offs with respect to exploration.
2 Related Work
Policy Representation Policies for continuous control are typically chosen to have continuous support and are commonly represented as Gaussians with diagonal covariance matrices [43, 16, 1, 18], Gaussian mixtures [56], Beta distributions [11], J. or latent variable models [15]. Exploration is then achieved by sampling from the continuous distributions with recent work relating maximum entropy objectives to the need for maintaining sufficient exploration in bounded action spaces [54]. Some approaches consider discretizing the action space facilitate exploration [14], but need to address the underlying curse of dimensionality [3, 12, 34, 50, 58]. The work by Tang and Agrawal [48] is the most closely related our work showing that action space discretization can yield state-of-the-ard performance for on-policy learning based on sufficiently large number of discrete components. Here, we provide insight into how even the most extreme discretization of only bang-bang actions can yield state-of-the the-art performance for variety of on-policy and off- tefl-plicy algorithms.
Bang-Bang Policies Bang-bang control problems have been considered very early on in reinforce- ment learning [53, 30, 2]. These early works mainly consider problems known require such policy type. While it was often observed that bang- bang-bang policies naturally emerge [21, 41, 51] most literature focused on avoiding this behavior [7, 19, 25]. while some leveraged it for specific applications [23, 26]. In contrast, our work focuses on understanding its nature, particularly in scenarios that were assumed to require continuous actions for obtaining an optimal controller.
Bang-bang controllers have been extensively investigated in optimal control research. Early works date to the 50s and 60s [45,4] 4] when it was shown that bang-bang policies arise as optimal controllers in minimum time problems [32, 31]. More recent work in bang-bang control focused on, e.g., further analysis [8, 55), numerical computation [9, 38, 39], and consideration of more specialized systems [10, 29, 35, 57]. Another related line of work are switched dynamical
Relation to Control
2

---

systems where each mode corresponds to a bang- bang-bang configuration. In these settings, one typically optimizes for switching times given a mode sequence [47] or penalizes the switching frequency as the control effort [46]. Unlike our work, research in this space typically does not consider the dynamics of the learning process deep RI. setting. However, we can still leverage this existing body of work to inform our explanation for the seemingly surprising emergence of bang-bang policies in RI..
3 Optimal Control with Continuous-Time Deterministic Dynamies
We start by describing connections between the emergence of bang-bang behavior in optimal control and common continuous control reinforcement learning problems. First, characterize three common reward structures together with the optimal controllers they induce. Then, we frame the solutions and their underlying assumptions in the context of our reinforcement learning setting.
We consider continuous- time deterministic dynamical system with state R" and action a E R. The continuous time-setting simplifies analysis and provides good approximation under the high sampling rates that are common on continuous control benchmarks and real-world robotic system. We define the non-linear control-affine system dynamics together with the objective function as
maximize S. r(s(()) - c(a(t))dl,
*(1) -0)0)((())) + (({())a((), OSIST.
S, is the time, I' is the final time, : + IR is the state reward and e: A IR the action cost. We assume action a(t) be scalar in the set A {a(t) R [a(t) = 1}. The following derivations can be directly extended to multidimensional inputs [27, Sec. 5.2].
where f
In optimal control settings, this is commonly solved by formulating the corresponding Hamiltonian
H (ss(t), a(t), = r(s(t)) cla(t)) + (atpt(((((((t)))))aappppp
g(s(())a(t)),
where p(t) is the costate variable, and leveraging Pontryagin's maximum principle [6, Prop. 3.3.1][27, Sec. 5.4] to derive the necessary optimality condition for action at (() E A and < 7 Tas as
H (ss (t), (t), p0)pp)(())) 2 H(s (t),
va(t) E A.
Maximum state reward (MS) Consider the case when c(a(t)) 0. Based on Pontryagin's maximum principle, the optimal control determined by solving the following simple linear program
maximize p²))) g(s" (f))((t)
subject to -1 a(t) S 1.
Based on the optimization surface in Figure (left) we see that the optimal action is given by
p" p²(1) ((t)) <0 9 =0
a" (t)
undetermined
p" (t)) (t))) (t)
If the undetermined condition does not occur, this is referred to as bang-bang control.
Minimum fuel-type cost (MF) Consider the case when c(a(t)) = |a(t). Through a slight refor- mulation under introduction of optimization variable 2 we arrive at another simple linear program
maximize
= = p+ (() (()))((s(())(())
subject
= = a(l) -z
-15 a(t) 1.
Based on the optimization surface in Figure (right) we see that the optimal action is given by
p" (1)T <-1
a" a*(1)=
p" (t)F 1p* (t)g(s
undetermined
If the undetermined condition does not occur, this yields bang-off-bang control (see Appendix A).

---

)(5))))
x² g(st) >
p¹ 0(8*) <0
p² g(*)) > 0
all)
a(t)
bang-bang solutions with arrows representing the cost function coefficient p" (t)² g(s* (t)). The feasible region is the line segment [-1,1] 1]. Right: Bang-off-bant solutions with
Figure 1: Left:
arrows representing the cost function gradients (1, (t)² g(("s (t))) and feasible region in light blue.
Minimum energy-type cost (ME) Consider the case when c(a(t)) a(t)?. In general, this formulation leads to non bang-bang optimal control [6, Example 3.3.2].
Singular ares and chattering behavior In the undetermined conditions the Hamiltonian is no longer dependent a({) and the necessary condition does not allow for determining a ((). These cases are commonly referred to as singular arcs. A sufficient condition to avoid singular ares is having a linear time-invariant dynamical system that is controllable [27, Sec. 5.6]. If an optimal control problem has singular arcs, a bang-bang solution may have to switch infinitely often in finite time interval achieve the desired state (t). This behavior is referred to chattering [37] or Zeno's 's phenomenon [59]. It does not appear in our formulation as we are considering discrete-time dynamical systems with a finite number of discretization points.
Discretization effect The Pontryagin maximum principle cannot be directly extended discrete- time dynamical systems without introducing additional assumptions. For example, Bertsekas [6. Prop. 3.3.2) derives discrete time version which, in order to provide necessary optimality conditions, requires convexity of the set A. If we consider only bang-bang actions, A is not convex and the discrete-time derivations no longer hold [6, Example 3. 10]. Nevertheless, find that our discretiza- tion intervals are sufficiently small to approximate the continuous case (see also Appendix C).
Stochastic dynamics This simplified setup can be seen as a special case of reinforcement learning problems because of its deterministic dynamics. However, similar derivations can be done in of stochastic action policies and dynamics. For example, this is the case when s(1) = [(((() + g(s(())a()) + w(),, where w())= IR" is random disturbance. Stochastic dynamics, together with the time discretization, are the key components to bridge this setup and the standard RI. frameworks, which are discussed in the next section.
Optimal controller Solving optimal control problems using the Pontryagin maximum principle is, in general, very challenging since it requires the numerical solution of a set of partial differential equations [6, 27]. Instead, we directly learn switching controllers stochastic Bernoulli policies.
Reinforcement Learning Preliminaries
We formulate the control and reinforcement learning problems in the context of Markov Decision Process (MDP) defined by the tuple (S,A, T,R. R. where and denote the state and action space, respectively, T: S x A + S represents the density of the transition distribution, S x A -> IR the reward mapping, and 7 E [o, 1) is the discount factor. Note that this reward can obtained from the previous section on continuous time dynamics by defining R(s,, a;) = f r(s(7)) c(a(T))dr. We define St and to be the state and action at time t, with input constraints of the form jaz| < amoz. Let To (a]s (s)s denote an action distribution parameterized by 0 representing the policy and define the discounted infinite horizon return G, Σ -* ag)) where 81+1 ~ T( 1st, ax) and a, wa (ag(s)). The objective is to learn the optimal policy maximizing the expected infinite horizon return G.] under unknown dynamics and reward mappings. This can be achieved by modeling (a, (x)) as distribution with neural network predicting the corresponding parameters 0 from st.

---

Domain
Cartpole
Cartpole
Cheetah
Dog
Finger
Humanoid
Quadruped
Walker
Task
Swingup
Sparse
Run
Walk
Spin
Walk
Run
Walk
Type
MS&ME
MS
MS
MS
MS
MS&ME
MS
MS
Table 1: DeepMind Control Suite tasks with their objective types based on Section 3.
Algorithm
Model
Learning
Inputs
Gradient Estimation
PPO [43]
model-free
on-policy
states
REINFORCE
SAC [16]
model-free
off-policy
states
Reparametrization
MPO [1]
model-free
off-policy
states
Expectation Maximisation
Dreamer V2 [18]
model-hased
off off-policy
images
REINFORCE. & Reparamctrization
Table 2: Algorithms analyzed in conjunction with Bernoulli policies.
Experiments
In this section, we work improve our understanding of empirical performance and characteristics of controllers learned via RL in continuous control domains. We compare performance and learnt distributions for Bernoulli and Gaussian policies on several tasks from the DeepMind Control Suite [49] based on a variety of popular RL algorithms. We then analyse the entanglement of exploration and converged solution in RL settings and evaluate robustness of the learned policies. We select MPO for this analysis as it does not require modifications to handle discrete distributions. First, we focus on the final solution by distilling Bernoulli policies from Gaussian teacher. Next, we discuss advantages and disadvantages of bang-bang behavior for exploration. Finally, we evaluate the effects of augmenting the objective by action penalties on exploration and final solutions of Gaussian and Bernoulli policies to demonstrate the trade-off's when mitigating bang-bang behavior.
5.1 Algorithms
We consider Proximal Policy Optimization (PPO) [43], Soft Actor Critic (SAC) [16], Maximum Posteriori Policy Optimization (MPO) [1] and DreamerV2 [18] as our baseline algorithms. These approaches differ widely in their learning method, optimization strategy, and input signals. A brief comparison of their characteristics is provided in Table 2 with additional description in Appendix B. We investigate the effects of replacing the commonly used Gaussian with Bernoulli distribution and provide both qualitative and quantitative comparisons of the learned controllers. For approaches requiring reparameterization of Bernoulli policies we leverage the biased straight-through gradient estimator [5] (Appendix B). We place particular focus on MPO robust off-policy method which does not require gradient reparameterization and avoids introducing additional estimation errors.
5.2 Solving Continuous Control Problems with Bang-Bang Policies
We investigate performance of Bang-Bang policies on several continuous control problems from the DeepMind Control Suite. Figure 2 provides learning curves for PPO, SAC, MPO, and DreamerV2, We note that restricting agents to only select minimum or maximum actions generally does not prevent learning on the tasks and even yields competitive performance. This applies across several domains and distinct algorithm designs (see Table 2).
The presence of action penalties reduces performance of the Bernoulli policies even when the resulting state-space trajectories appear optimal. For instance, bang-hang control is able to swing-up and stabilize on Cartpole Swingup but incurs steady-state cost during stabilization as it is unable to select magnituie actions. We also observe lower performance of Bernoulli policies for SAC on Cartpole tasks. SAC presents some unique challenges, the target entropy parameter operates on different scales in the continuous and discrete distribution cases. We found that particularly domains with din-esenniioall action spaces, such as Cartpole, can suffer from premature policy convergence. Finally, bang-bang control with Dreamer V2 does not perform well on Finger Spin. The continuous policy already displays some difficulties in learning the task comparison PPO, SAC, and MPO. This could indicate that learning latent model from images can be challenging on this task. Building the model under only extreme input signals potentially exacerbates these effects. Quantitatively, we observe strong performance when restricting the agent employ Bernoulli policies.
5

---

Campeee Sammup
Campole Spare
Chretan Bun
Reger Spin
Wekker Was
Duwerrpeed Run
Himanoit War
Day Wal
2003
Epiiode
20
60
26
48
98
30
40
80
24
M
15
360
31
63
Epicdde Sacum
Epritdde Ratumn
Epiiede Saturn
02
Figure 2: Comparison of Bang-Bang and Gaussian policy heads for solving continuous control tasks. We evaluate PPO, SAC, MPO, and DreamerV2 on several domains from the Deep Control suite. Generally, find that the Bang-Bang policies perform par with the normal Gaussian policies.
Cartzole
Watser
Qundraped
Humanoid
Finger
-3
-3
-1
-1
Sting Stnng
Figure 3: Distribution of actions along trajectory for MPO. We consider 11 bins per action dimension and aggregate over 1000 steps. The Gaussian policy exhibits bang-bang behavior several domains. Action penalties, particularly Cartpole Swingup (top left), reduce bang-banp action selection.
The results in Figure 2 suggest that across most domains, we do not require a continuous action space. To investigate this phenomenon, analyze the action distributions of converged policies. Figure 3 provides binned actions aggregated over an evaluation episode for converged Bernoulli and Gaussian policies in several domains. We that the Gaussian policies consistently resort to extremal actions, particularly on the Finger, Walker and Quadruped tasks. The Cartpole behavior splits into phases, leveraging bang- bang-bag control for fast swing-up and near-zero actions during stabilization. Qualitatively, we find that the Gaussian policies tend to converge to bang-ban behavior.
5.2.1 Disentangling Exploration and Final Solution
The trained Bernoulli and Gaussian policies display strong similarities in their quantitative and qualitative performance. Since the final performance is affected by both exploration and the ability to represent the optimal controller, the Bernoulli policies could compensate for lack of expressivity by enabling better exploration. To focus on the final solution and further investigate similarities between converged policies, we leverage behavioral cloning to investigate the extent which Bernoulli policy is able to learn from a Gaussian teacher. The student acts in the environment while being supervised by converged Gaussian. We maximize log-probability over actions additionally discretize Gaussian targets for the Bernoulli student. Figure 4 provides learning curves for both Bernoulli and Gaussian students. Strong similarity between curves indicates strong bang-bang characteristics of the Gaussian teacher. The Bernoulli policy performs surprisingly well all but the

---

Cortooo Swingup
Carpole Sparse
Cheecah Run
Riger Soin
Walker Wark
Guddruped Run
Humarod Work
Dog Walk
Tamestep
Timeetep [S]
Timestpp [N]
Timanbep [M]
Timssopp (M]
Timesop
Timestep [N]
TImmstep
Bang Sang
Caras
Figure Behavioral cloning with Gaussian teacher. The Bang- Bang and Gaussian students perform similarly well across several domains, further indicating that the teacher leverages bang-bang control.
Humanoid task. This indicates that the Gaussian teachers consistently leverages bang-bang action selection and differences from bang-hang behaviour have only limited impact on performance.
5.3 Robustness Under Perturbations
We observe similar converged performance of the Bernoulli and Gaussian policies. An argument against bang-hang control for articulated robotic systems from controls perspective can be stability, particularly in the absence of ideal sensing or actuation delays that may trigger overshooting. We therefore investigate the impact of sensor degradation and modification of the environment parameters based on the Real-World RL Challenge Framework [13].
Environment Modifications To assess whether our observations generalize more challenging task variations we consider modifications of the environment parameters. Specifically, we investigate performance under changes to the system mass, agent morphology, and friction or damping char- acteristics across the Cartpole, Walker, and Quadruped domains. Figure (left) provides learning curves for MPO with Bernoulli and Gaussian policies, where parameter values are provided at the top of each subplot in order. We note that both policy types perform comparably across all tasks and parameter values, indicating that the Bang-Bane Bang policy is robust to variations of the environments.
Transfer under Disturbances We evaluate policy transfer under real-world inspired sensor degra- dation. We consider reduction of the control frequency. stuck sensor signals, dropped sensor signals, sensor delay, and sensor noise (details Appendix E). The Gaussian action space is superset the Bernoulii action space, yiclding richer set of interactions that could translate to better generalization around the optimal trajectory. Figure (right) provides normalized scores relative to performance under ideal sensing based on 40 trajectories. We find that the Bernoulli and Gaussian policies display similar performance. This indicates that robustness to disturbances is not negatively affected by limiting controls bang-bang within the environments considered. A potential reason for this is that the robot dynamics act as a low-pass filter on the bang-bang input to effectively smooth input jumps.
5.4
Characteristics of Bang-Band Control in RL settings
Learning bang-bang constrained policies can yield competitive performance that generalizes across several environment formulations and algorithm designs. Next, discuss some characteristics of bang-bang control in RL settings and evaluate methods to avoid such behavior in Gaussian policies.
5.4.1 Intrinsic Exploration of Sparse Rewards
Pointmass Bang-bang policies only apply extremal actions. By forcing highest magnitude actions, the intrinsic exploration characteristics of an agent are affected. Figure provides two tasks for point-mass controlled in x-y position. Each task is evaluated with no or quadratic action penalties (left, right). Here, consider Bang,Bang, Bang-Off-Barn the agent is always initialized at the center and does not receive any rewards. The Bang-Bany policy exhibits the largest area coverage by virtue of only sampling large magnitude actions. This effect magnified under action cost, as other agents limit exploration to reduce cost (see also Appendix F). In bottom row, the agent starts at the center without resets and receives sparse rewards at the circles. The Bang-Bang policy again exhibits the largest coverage. While its large magnitude sampling facilitates escapes from local optima, can also impact stabilization at global optima (Figure 6, right). Overall, Bang- Bang policies offers strong intrinsic exploration that is robust to action penalties.
and Gaussian policies. In the top row,
Bang-Bang
7

---

Thirhg Peturtaton
Tornfer under Denumanees
Pole Masa 18.00 1
seeet 10.0 1661
Sioer Dumann 32-44.311300 31x0
Cerooee
500
Mack
Naat Scale 35, 2,000
Largh 18.6 0.53)
Pretin 0301
3025, 2,00)
Gremec L5g)
Qhatht Cpass S011
16
Figure 5: Robustness of the Bang-Bang policies when considering task modifications based on the Real-World RL suite [13]. Left: training with perturbed environment parameters. Each subplot compares the effect of changing one parameter on training performance. Right: transfer under disturbances, performance normalized with respect undisturbed scenario. Bang- Bang and Gaussian policies perform comparably, indicating similar robustness in simulation.
(- 0.34
Initial con llton
Sang Bang
Gaussian
Bang Sang
Bang OffiBang
Gassskan
Figure 6: Pointmass exploration tasks. Top: the agent is reset every episode with no reward feedback. Bottom: the agent always continues and receives different sparse rewards as indicated by the values in the circles. Bang-Bang control leverages extreme actions yielding strong passive exploration, unaffected by action penalties (top). This can facilitate escape from local optima and impede stabilization at global optima (bottom). In sparse exploration tasks, action penalties may prevent the agent from sufficient exploration when it is able to focus on minimizing action cost instead.
Control Suite While action penalties limit maximum performance of Bang-Bang policies, they can also negatively affect exploration Gaussians as observed Figure 6. Exploration is hindered whenever reward maximization is traded-off to minimize action cost. This behavior favored by sparse feedback, as the agent needs to actively explore to observe rewards. We compare performance of Bang-Bang, Lang-Off-Bann
and Gaussian policies three versions of the same tasks: dense, sparse, and sparse with action cost (details on reward sparsification are provided in Appendix D). Figure 7 provides learning curves for the Cartpole Swingup, Walker Walk, and Quadruped Run domains. All three policy parameterizations perform similarly on the dense versions of the tasks. The sparse Cartpole and Walker tasks without action cost yield similar converged performance for all policies, while the Lang-Bang, policy converges faster due to its strong passive exploration. On the sparse tasks with action penalties, both the Gaussian and Bang- Lonn-Off-Bii Off-Bang controllers are unable to solve the task and focus on action cost minimization instead of exploring sufficiently in search of reward (convergence to 0 returns). The Bang-Bany controller solves these tasks with steady-stan -tate action cost, it only selects maximum magnitude actions and effectively ignores the detrimental impact of action penalties on exploration. This example further highlights the intricate interplay between finding policies that avoid bang-bang behavior and enabling sufficient exploration.

---

Carppole Swingup
Walker Walk
quodraped Run
142
125
DonGo
Eptrode
Heward Dractaa Druutar
Volee Denelly
Episode
Timesteo (M]
Timestep
(M)
Timestop [N]
Figure 7: Exploration under sparse reward feedback and action penalties for Bang-Bang Bang- Off- Bang, and Gaussian policies. While all three policy types perform comparably on the dense task versions (column 1), Bang-Bang policies can yield faster convergence on sparse tasks due to passive exploration arsing from extremal action selection (column 2). Furthermore, large action penalties can limit exploration in favor of action cost reduction for the non-Bang Bang policies (column 3).
expet
f2- 0.1a*
Value Reration
Bang Bang
Gaussian
Performance
Value Iteration
Bang Bang
Gaussian
Performance
Causiiun
Timesteps (M)
Figure 8: Learning to swing-up pendulum under no action cost (left) and quadratic cost (right). The Bang-Bang policy converges faster without action penalties, the Gaussian yields higher returns with action penalties. Learned policy mappings are overlayed with sample swing-up trajectories (black).
5.4.2 Action Penalties
Bang-Bang policies enable fast exploration and can arise optimal controllers in the absence of action cost. However, robotics applications may suffer from extensive extremal action selection and action penalties can mitigate bang-bang behavior. We highlight these considerations on a pendulum swing-up task with no and quadratic action cost in Figure 8. Without action cost (left), both Bang- Bang and Gaussian policies learn to approximate the optimal action mapping and solve the task with the Lang-Bang policy offering a faster rise time. With action cost (right), representing the minimum energy category in optimal control (Section 3), the Bang-Bang policy is unable to represent the optimal action mapping and continues to incur steady-state cost (see also Appendix F).
We investigate the effect of augmenting the reward function of the more complex Walker and Quadruped tasks with action penalties. Here, we consider both quadratic cost to reduce action magnitude and difference cost increase smoothness. In the latter case, we augment the system state by the previous action. Figure! (top) displays Gaussian means along a Quadruped Run trajectory for the different penalty structures and their combination. We observe the desired effect of lower magnitude and smoother transitions in the learned distributions. Figure 9 (bottom) provides training curves under the different reward structures, performance based on the nominal reward function, and transfer under disturbances (see Section 5.3) for the Walker (left) and Quadruped (right) tasks. We note that while action penalties help to mitigate bang-bang behavior, robustness of the resulting gaits is only slightly increased. Furthermore, the modified reward structure may yield reduced performance of trained agents as measured by the nominal reward function and observed on the Quadruped. These findings suggest that while potentially undesired bang-bang action selection can be reduced by introducing action penalties, naive engineering of the reward structure may force the agent into sub-optimal behaviors. The question then becomes what metrics to consider when evaluating agent performance in light of transfer real-worla systems. Pure return maximization may exploit both simulation and task peculiarities and is not sufficient for comparing the merit of different algorithms.

---

Action ma?. ca-aa Aar)
Q-- Bi Counts
Walser Wait
Quscnipec San
braningg
oction
Depliymment unoer duturbenoee
Meal bunnng
esttio cost
Deployment under distrrbnncs
Folicce beturn
Figure 9: Impact of action penalties. Top: Gaussian means under varying penalties along a Quadruped Run trajectory. Bottom: training and transfer performance on Walker (left) and Quadruped (right). While action penalties help to mitigate bang-bang behavior, the resulting gaits only slightly increase robustness and may reduce performance as measured by the nominal reward function (Quadruped).
6
Discussion
In this paper, we investigate the emergence of bang-bang behavior in continuous control reinforcement learning and show that several common benchmarks can fact be solved by bang-banp constrained Bernoulli policies. We draw theoretical connections to bang-bang solutions from optimal control and perform extensive experimental evaluations across variety of state-of-the-ar RL algorithms. In particular, we theoretically derive when bang-bang control emerges for continuous-time systems with deterministic dynamics and describe connections to continuous control RL problems considered here.
In our experiments, we find that many continuous control benchmarks do not require continuous action spaces and can be solved (close to) optimally with learned bang-bang controllers for several recent model-based and model-free, on-policy and off- off-policy algorithms. We further demonstrate the efficiency bang-bang control in more realistic variations of common domains based on the Real-Worle RL suite [13] and demonstrate how action costs affect the optimal controller. Generally, Bernoulli policies may be applicable when dynamics act as a low-pass filter to sufficiently smoothen the switching behavior. A key advantage can then be the reduction of the policy search space |A|, while the resulting extreme actions can favor coarse exploration. Since exploration, learning and the final solution are entangled in RL, we perform additional experiments with focus on behavioural cloning of a trained Gaussian controller to compare the performance of bang-bang and continuous policies without the additional challenges arising from exploration. It is important to note that we still require sampling -aasee optimisation, which has become popular to handle continuous action spaces. However, the main reason is often not the continuous action space but the potential high-ding dige-niimesiinlltty of the action space which would otherwise become intractable.
from R.N to 2 with
Benchmark design is complex task with the goal of measuring progress that is transferable to impactful real-world applications. We demonstrate that the learnt controllers for continuous control benchmarks without action penalties can exhibit bang-bang properties which can be damaging to real-world systems. At the same time, we also show that including action penalties can significantly impact exploration for Gaussian policies. Integrating the impact of task design on both exploration as well as final solution properties and investigating algorithms which can overcome local optima while still learning smooth controllers is important direction for future work.
Societal Implications: Improved understanding of emergent bang-bang behavior may broaden real- world applicability of RL policies. While resulting approaches could be used in ways not intended by researchers there are many beneficial applications such as robotics for improved productivity and workplace safety. Even desirable outcomes can come with side-effects such as job loss and societal transformation costs. While the authors ultimately believe that societal benefits of this work outweigh its harms, these considerations need to be re-evaluated on constant basis as applications emerge.
10

---

Acknowledgments and Disclosure of Funding
Tim Seyde, Igor Gilitschenski, Wilko Schwarting and Danicla Rus were supported in part by the Office of Naval Research (ONR) Grant N000 100011418-112833 and Qualcomm. This article solely reflects the opinions and conclusions its authors and not any other entity. We thank them for their support. The authors further would like to thank Lueas Liebenwein for assistance with cluster deployment, and acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources. We would also like to thank Murad Abu- Khalaf for insightful discussions well as the NeurlPS reviewers and program chairs for their helpful feedback and suggestions for improvement.
References
[1] A. Abdolmaleki, J. T. Springenberg. J. Degrave, S. Bohez, Y. Tassa, D. Belov, N. Heess, and M. Riedmiller. Relative entropy regularized policy iteration. arXiv preprint arXiv:1812.02256,
2018. 2, 14
[2] C. W. Anderson. Learning to Control an Inverted Pendulum with Connectionist Networks. In Proceedings of the American Control Conference (ACC), 1988. 2
[3] R. Bellman. Dynamic programming. Technical report, RAND CORP SANTA MONICA CA, 1956.
[4] R. Bellman, I. Glicksberg, and O. Gross. On the "bang-bang" control problem. Quarterly of Applied Mathematics, 14(1), 1956. 2
[5] Y. Bengio, N. Léonard, and A. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv::308...4 3432, 2013. 5, 15
[6] D. Bertsekas. Dynamic Programming and Optimal Control, volume 1. 4 edition, 2005. 3,
[7] Bohez, A. Abdolmaleki, Neunert, J. Buchli, N. Heess, and R. Hadsell. Value constrained
model-free continuous control. 1902vv1192..46222 2019. 1.2
[8] E. Casas, D. Wachsmuth, and G. Wachsmuth. Sufficient Second-Order Conditions for Bang- Bang Control Problems. SIAM Journal on Control and Optimization, 55(5), 2017. 2
[9] E. Casas, D. Wachsmuth, and G. Wachsmuth. Second-Orden Analysis and Numerical Approxi- mation for Bang- Bang-Bang Bilinear Control Problems. SIAM Journal on Control and Optimization, 56(6), 2018. 2
[10] N. Chen, Y. Wang, and D.-H. Yang. Time-varying bang-bang property of time optimal controls for heat equation and its application. Systems & Control Letters, 112, 2018.
[11] P.W. Chou, D. Maturana, and S. Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. In International conference on machine learning. pages 834 843. PMLR, 2017. 2
[12] Y. Duan, X. Chen, R. Houthooft, Schulman, and Abbeel. Benchmarking Deep Reinforce- ment Learning for Continuous Control. In International Conference on Machine Learning. (ICML),2116 2
[13] G. Amold. N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empirical investigation of the challenges of real-world reinforcement learning, arXiv preprint arXiv:2003. / /88, 2020. 7, 8, 10, 16
[14] G. Farquhar, L. Gustafson, Z. Lin, S. Whiteson, N. Usunier, and G. Synnaeve. Growing action spaces. In International Conference on Machine Learning (ICML), 2020. 2
[15] S. Fujimoto, D. Meger, and D. Precup. Off-Policy Deep Reinforcement Learning without Exploration. In International Conference on Machine Learning (ICML), 2019. 2
[16] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018. 2,
[17] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, and S. Levine. Soft actor-critic algorithms and applications. Technical report, 2018 14
11

---

[18] D. Hafner, T. Lillicrap. M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020. 2, 14
[19] B. J. Hodel. Learning to Operate an Excavator via Policy Optimization. Procedia Computer Science, 140, 2018. 2
[20] M. Hoffman, B. Shahriari, J. Aslanides, G. Barth--Mroo, F. Behbahani, T. Norman, A. Abdol- maleki, A. Cassirer, F. Yang, K. Baumli. S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcchre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006. 00099, 2020. 14
[21] S. H. Huang, M. Zambelli, Kay, M. F. Martins, Y. Tassa, P. M. Pilarski, and R. Hadsell. Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning. 1rXiv:1903.08542. 2019. 1,2
[22] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 1611
2016. 15
[23] W. Jaškowski, O. R. Lykkebø, N. E. Toklu, F. Trifterer, Z. Buk, J. Koutník, and F. Gomez. Reinforcement Learning to Run... Fast. In S. Escalera and M. Weimer, editors, NIPS 2017 Competition Book. Springer, Springer, 2018.
[24] M. Johnson, B. Shrewsbury, S. Bertrand, T. Wu, D. Duran, M. Floyd, P. Abeles, D. Stephen, N. Mertins, A. Lesman, et al. Team ihme's lessons learned from the darpa robotics challenge trials. Journal of Field Robotics, 32(2):192-2006 2015. 15
[25] E. Joos, F. Péan, and O. Goksel. Reinforcement Learning of Musculoskeletal Control from Functional Simulations. In Medical Image Computing and Computer Assisted Intervention
(MICCAI), 2020. 2
[26] t.. Kidziński, S. Mohanty, C. F. Ong. Z. Huang, S. Zhou, A. Pechenko, A. Stelmaszezyk, P. Jarosik, M. Pavlov, S. Kolesnikov, et al. Learning run challenge solutions: Adapting reinforcement learning methods for neuromuseulloskelttal environments. In The NIPS'17
Competition: Building Intelligent Systems, pages 121-153. Springer, 2018. 2
[27] D. E. Kirk. Optimal control theory: an introduction. Courier Corporation, 2004. 3. 14
[28] A. B. Kordabad, W. Cai, and S. Gros. MPC-based Reinforcement I Learning for Economic Prob- lems with Application to Battery Storage. In Proceedings of the European Control Conference (ECC), 2021.
[29] K. Kunisch and L. Wang. The bang-hang property of time optimal controls for the Burgers equation. Discrete Continuous Dynamical Systems, 34(9), 2014.
[30] Lambert and M. Levine, A two-stage learning control system. Trans. on Automatic Control, 15(3), 1970. 2
[31] J. Lasalle. The 'bang-bang' principle. IFAC Proceedings Volumes, 1(1):503-507, 1960. ISSN 1474-6670. doi: http://0...ogg/0000000664177677- / //www/sc sci sciencedirect. wwtirreettcooooeee com/ http/sccenceeeee wwweenneeattiiiiieeeeeeee p tional IFAC Congress on Automatic and Remote Control, Moscow, USSR, 1960. 1,2
URL https: 1st Interna-
[32] P. LaSalle. Time Optimal Control Systems. Proceedings of the National Academy of Sciences, 45(4), 1959. 2
[33] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), 2020. 15
[34] T. P. Lillierap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016. 2
[35] J. Lohéac and M. Tucsnak. Maximum Principle and Lang-Bang Property of Time Optimal Controls for Schrödinger-Tyee Type Systems. SIAM Journal on Control and Optimization, 51(5),
2013. 2
[36] C. J. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint [to::1611.0012. 2016. 15
[37] I.. Manita. Optimal operating modes with chattering switching in manipulator control problems. Journal of Applied Mathematics and Mechanics, 64(1), 2000. 4
12

---

[38] H. Maurer, C. Büskens, J.-H. R. Kim, and C. Y. Kaya. Optimization methods for the verification of second order sufficient conditions for bang- bang controls. Optimal Control Applications and Methods, 26(3). 2005. 2
[39] A. Münch and F. Periago. Numerical approximation of bang-bang controls for the heat equation: An optimal design approach. Systems & Control Letters, 62(8), 2013. 2
[40] R. Munos, T. Stepleton, A. Harutyunyan, and M. G. Bellemare. Safe and efficient off-policy reinforcement learning. arXiv preprint arXiv: 02647, 2016. 14
[41] G. Novati and P. Koumoutsakos. Remember and Forget for Experience Replay. In International Conference on Machine Learning (ICML), 2019. 1,2
[42] F. Pardo. Tonic: A deep reinforcement learning library for fast prototyping and benchmarking. arXiv preprint arX1v:2011.07537, 2020. 14
[43] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 1707vv:70..06333
2017. 2,5, 14
[44] T. Seyde, W. Schwarting. S. Karaman, and D. Rus. Learning to plan optimistically: Uncertainty- guided deep exploration via latent model ensembles. arXiv preprint arXiv:2010. 14641, 2020. 15
[45] L. M. Sonneborn and F. S. Van Vleck. The Bang- Bang Principle for Linear Control Systems. Journal of the Society for Industrial and Applied Mathematics Series A Control, 2(2), 1964. 2
[46] B. Stellato, T. Geyer, and Goulart. High-speed finite control set model predictive control for power electronics. JEEE Trans. on Power Electronics, 32(5), 2017. 3
[47] B. Stellato, S. Ober- Blöbbum, and P. Goulart. Second-order switching time optimization for switched dynamical systems. IEEE Trans. on Automatic Control, 62(10), 2017.
[48] Y. Tang and S. Agrawal. Discretizing Continuous Action Space for On-Policy Optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[49] Y. Tassa, S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez,
T. Lillicrap, and N. Heess. dm con_col:o:: Software and tasks for continuous control, 2020. 5
[50] A. Tavakoli, F. Pardo, and P. Kormushev. Action Branching Architectures for Deep Reinforce- ment Learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018.
2
[51] T. G. Thuruthel, E. Falotico, F. Renda, and C. Laschi. Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators. IEEE T-RO, 35(1), 2019. 1, 2
[52] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. 14
[53] M. Waltz and K. Fu. A heuristic approach to reinforcement learning control systems. IEEE TACON, 10(4), 1965. 2
[54] C. Wang. Y. Wu, Q. Vuong, and K. Ross. Striving for simplicity and performance in ooflpo drl: Output normalization and non-uniform sampling. In International Conference on Machine Learning, pages 10070-10080. PMIR, 2020. 2
[55] G. Wang and Y. Zhang. Decompositions and bang-bang properties. Mathematical Control & Related Fields, 7(1). 2017. 2
[56] M. Wulfmeier, A. Abdolmaleki, R. Hafner, J. T. Springenberg, M. Neunert, N. Siegel, T. Her- tweck, T. Lampe, N. Heess, and M. Riedmiller. Compositional Transfer in Hierarchical Reinforcement Learning. In Robotics: Science and Systems (RSS), 2020. 2
[57] D.-H. Yang, B. Guo, W. Gui, and C. Yang. The Bang-Bang Property of Time-Vary Varying Optimal Time Control for Null Controllable Heat Equation. Journal of Optimization Theory and Applications, 182(2), 2019. 2
[58] Y. Yue, Y. Tang, M. Yin, and M. Zhou. Discrete action on poloiic learning with vallo critic. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2020. 2
[59] Zhang, K. H. Johansson, J. Lygeros, and S. Sastry. Zeno hybrid systems. International Journal of Robust and Nonlinear Control, 11(5), 2001. 4
13


Reviews:
[{"review_id": "sPSw0oRAHSB", "reviewer": "Reviewer_48Th", "summary": "The paper deals with the question namely the issue when an optimal\ncontinuous (\"soft\") control can be superseded/replaced or even needs\nto be implemented by a Bang-Bang control.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper deals with an interesting question, namely the nature of\noptimal control solutions and whether a simplistic bang-bang might\nwork just as well as a continuous (e.g. Gaussian) control. The reviewer\nparticularly likes the idea to ask what the nature of the problems\nwould be which would favour bang-bang, rather than just presenting\n\"yet another algorithm\" - in this sense, this paper is researching the\n*nature* of problems leading to particular optimal effects. The paper\nalso is careful to note that RL entangles exploration, learning and\nexecution, and the issue of trade-offs. Amongst other, action costs\nmay discourage bang-bang, but for Gaussian exploration they may\nconfine the exploration; it is clear that there are many subtleties\nthat dictate the preferred outcome.\n\nA very compact derivation (based on Pontryagin) and discussion of\nemergence of bang-bang and a set of examples where bang-bang is\npredominantly seen in a number of \"walking\"-type scenarios.\n\nA number of instructive analyses demonstrate the prevalence of various\nsolutions and dynamics under varying action costs and thus balances\nand thus sheds light on the nature of problems and cost structures\nthat favour bang-bang against soft controls. An interesting research\npaper on the meta-question of what solutions one can expect for\ncertain problem formulations.\n\nI have no particular comments on this well-written paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations etc. are very well discussed throughout.\n\nAs for Societal Impact, this is essentially basic research; there should be an N/A option on the \"Societal Impact\" statement. Asking authors of basic research to comment on societal impact, unless it is directly relevant is no better than asking them to read entrails.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h 15m"}, "extra_text": {}}, {"review_id": "VaKNYJ790ns", "reviewer": "Reviewer_39gS", "summary": "The paper focuses on the so-called bang-bang policies. These are ones, which are allowed only for discrete values, typically two extremal ones. Even though they are usually disadvantageous in real-world scenarios, they tend to arise in simulation, and they seem to deliver at least decent performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper focuses on the so-called bang-bang policies. These are ones, which are allowed only for discrete values, typically two extremal ones. Even though they are usually disadvantageous in real-world scenarios, they tend to arise in simulation, and they seem to deliver at least decent performance. I find the paper overall interesting and valuable; it is all generally well written. It presents a theoretical analysis and proper empirical evaluation. Below I list the pros and cons and some (typically minor) recommendations and questions. \n\n\n\nPros:\n\n* Choices of algorithms covers well the spectrum of the current state-of-the-art\n* The authors attempt to make a deeper analysis than just observing the basic phenomenon\n* Related work seems fairly complete (I am not an expert though, see also the cons section)\n* The paper is overall well-structured, well-written and easy to follow (with exceptions mentioned in the cons section)\n* In general, I appreciate additional experiments \n\n\n\nCons:\n\n* The papers fall into a benchmark-like category. As such, there are higher expectations as to the empirical section. I realise that it is always a hard choice, and one needs to stop. Still, I'd appreciate having more environments. Moreover, the authors use $4$ seeds, which might be too small for RL settings. \n* I find Figure 1 somewhat overwhelming. I appreciate putting all the training data. However, I'd find a useful summary. \n* I am not quite convinced about the exploration section. It is almost by design that bang-bang spans a bigger region (which the authors mention). If the conclusion is that this is an issue for exploration, I'd appreciate algorithmic developments towards increasing exploration in the Gaussian setting.\n* Similar problems, though from a different perspective, were studied in https://arxiv.org/pdf/1910.02208.pdf. I'd suggest the authors should at least mention this in Related Word or, better, compare SOP with bang-bang SOP.\n* I find Action Penalty somewhat incomplete and incomprehensive (again, I am overwhelmed by Fig 7. )\n* (Nitpick) The authors do not specify technical details of the environments. It'd be nice to make the paper self contain. \n* (Nitpick) I'd appreciate more details about discretised SAC than Alg 2 (say what is bijector)\n* (Nitpick) It is perhaps good to mention that discretisation was superior in a popular Learning to run contest: \"https://arxiv.org/abs/1804.00361\"\n\n\n\nQuestions and suggestions:\n\n* I acknowledge the theoretical section. I'd appreciate some discussion on how much the studied model is relevant to the environments in the paper and real-world scenarios. \n* (Nitpick) I'd appreciate having more hypotheses, why bang-bang work so well.\n\n\n\nOriginality: The paper is mostly a benchmark paper. \n\nQuality: Yes, this is a fair empirical work with some theoretical background.\n\nClarity: The paper is clear.\n\nSignificance: Comprehensive study of the bang-bang policies presented in the submission can be beneficial for the community. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I am not satisfied with the limitation description. Generally, I am fond of merging this with 'the further research discussion' (as the authors did). However, in this case, it is blurred to the point, I do not know what the authors think about the limitations of this paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8h"}, "extra_text": {}}, {"review_id": "RK7jfFVYLHM", "reviewer": "Reviewer_qvNS", "summary": "The work investigates emergence of bang-bang control policies in\ncontinuous RL problems, by making the connection to optimal control.\nExmpirically, it shows how continuous control problems are solved with\nbang-bang policies, using 4 different algorithms, and that for most of\nthe domains in these experiments, a continuous action space is not\nneeded.\nThe experiments also explore modified reward functions designed to\nincrease smootheness of the learnt policies and their effect on the\n(original) return obtained during exploitation.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the main contribution of the work is not a new method but\nrather a new investigation of existing techniques about emergence of\nbang-bang policies, with the goal to provide insights into designing\nRL experiments and rewards. Previous work (that is also referenced)\nfocuses on approaches helping to avoid such policies but there is no\nprevious investigation of the kind presented in this work.\n\nQuality: To my understanding the work is technicaly sound and supports\nthe claims, though is missing some details that would be good to\nsee. It consists of a theoretical analysis that provides an\nexplanation for emergence of bang-bang control, though without\t\ncomplete proofs in the main paper. The experimental part would benefit\nfrom more details (though parameters and some other details are\nprovided in the supplementary material): I may have missed it but\ndidn't see how often each experiment was run (or if each was only run\nonce). The work\tpresents a complete and carefully (and as far as I can\njudge honestly) considered investigation. It does leave\topen questions\nfor how to design rewards, benchmarks, or for how to avoid bang-bang\npolicies, and does not present a\tsolution to all\tthese, but rather\nprovides a basis for future work in these directions.\n\nClarity: The paper is clearly written and very well organised, and\npresentation including graphs is of high quality.\n\nSignificance: I do believe the results are important, and the work is\nset to be useful for future work in a number of directions - e.g., for\nlearning controllers where bang-bang policies a\tbetter avoided like in\nrobotics, or potentially also for approaches speeding up\nexploration. The work is not yet another paper improving\nstate-of-the-art on some benchmark (even though SOTA on some\nbenchmarks is achieved with the Bernoulli distributions used in the\nexperiments) but providing some new insights that I find more valuable\nbecause of wider implications.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are adequately addressed.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I have found no ethical issues with the paper.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BH8bPDdQGs", "reviewer": "Reviewer_wPsN", "summary": "The authors analyze the emergence of implicit bang-bang (action saturation) control behavior in Deep RL. They claim three contributions, i) they argue that these are often actually optimal by making a connection to time-optimal solutions in optimal control theory, ii) they show empirically that the performance of an inherently bang-bang (discrete action) control policy yields comparable performance, and iii) they discuss possible extensions to reduce bang-bang behavior and their connection to exploration . \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The emergence of undesired bang-bang behavior can be a problem in real-world applications since it can damage mechanical systems or cause instability. Any insight into this problem would be valuable for control applications. Clarity of presentation is adequate and the paper contains a wealth of empirical experiments which appear to be carefully documented.\n\nHowever, I have some concerns about the quality of the three main contributions as outlined below:\n1) The first contribution is a proof from optimal control theory that bang-bang is optimal in continuous-time minimum-state problems without action penalties. The theoretical contribution here seems rather minor since it appears to follow directly from a book on optimal control [23]. Anybody that has played around with MPC also won't be surprised that the controller will saturate any action limits if the action penalty is low enough.  \nHowever, shouldn't pure bang-bang technically only be *optimal* for continuous-time, where max control output can be countered by an arbitrary small time step? For discrete-time, if the state is close enough to the target, the optimal action could be anywhere in-between the action limits. The authors claim that this continuous-time behavior translates to discrete-time systems because the time steps in RL are generally small enough, but this is never properly motivated. Even without examining this, running at higher Hz could of course be a solution, but it requires more compute and that the state is also available at that rate.  \n The paper does raise a valid point about the importance of including action penalties in benchmark design, otherwise the agent will certainly saturate the action constraints, which may be undesirable in practice. Although the authors use the DeepMind control benchmarks as examples here, to the best of my knowledge, all the gym Mujoco benchmarks actually do include some kind of action penalty. I don't know if they are suitable or not. It also seems unlikely that bang-bang behavior observed in applications is just because they forgot to include an action penalty. This avenue is never really explored, and the paper never precisely defines what type of bang-bang behavior they are trying to fix - is it all kinds of action saturation, or just quick oscillations near an equilibrium? The latter may sometimes actually be desirable, but the latter never is.\n\n2) The second contribution is empirical experiments showing that a discrete-action (bang-bang) control policy achieves performance comparable to regular continuous Gaussian ones. However, this just proves that current methods with Gaussian policies perform as poorly as a bang-bang controller. If the Gaussian ones often exhibit bang-bang behavior as purported in the motivation to this paper, this result does not appear particularly surprising. Finally, this focuses on training curves, which is not the same thing as the final performance. The tracking error near an equlibrium will often be dominated by the penalties to get there, especially if a square norm is used. It is unfortunate that they never mathematically define what constitutes harmful bang-bang behavior. The only attempt at measuring this are the action distribution plots in Figure 2, but it is difficult to see what is spurious oscillation and action saturation due to design of the reward function (low action penalty). \n\n3) The third contribution is a purported discussion of possible extension to reduce bang-bang behavior by including action penalties. I didn't find anything supporting this except the recommendation to introduce action penalties. An important observation, but I suspect many practitioners with control background were already aware of this.  \nThe authors also argue that action penalties can impede exploration and try to disentangle these. I did not find this line of reasoning entirely convincing. Couldn't changing the action penalties also drastically change the optimal strategy (e.g. run vs. walk cycles are quite different), so even if the lack of action penalties frees it up to visit more of the state space, it may be learning in the wrong region of policy space (e.g. you really wanted an energy efficient walk, not a run)? Are plots comparing state coverage meaningful here?\n\nIn conclusion, the authors target an important problem but the contributions appear minor or vague. The main finding only seems to pertain to the benchmarks in the DeepMind control suite that lack action penalties. The paper fails to convince me that a simple lack of action penalties is the main reason why DeepRL policies sometimes exhibit harmful oscillations, in the real world or even in other common control benchmarks.\n\nSome details:\n\nL18-54 - Intro and motivation is a bit fuzzy with several claimed minor contributions\n\nL91: \"The continuous time-setting simplifies analysis and provides a good approximation under the high sampling rates that are common on continuous control benchmarks and real-world robotic system.\" - One would rather see the discrete time as an approximation to continuous time when it comes to control tasks.\n\nL112: \"Nevertheless, we find that in this work the discretizaton intervals are small enough to not affect optimality of the learned policies.\" - based on what?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The authors' answers have been appreciated by the reviewer, who reached a consensus about accepting this paper.
I want to congratulate the authors and invite them to modify their paper following the reviewers' suggestions.
Decision: Accept (Poster)