Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies

1 Introduction . Real-world robotics tasks commonly manifest as control problems over continuous action spaces . When learning to act in such settings , control policies are typically represented as continuous probability distributions that cover all feasible control inputs - often Gaussians . The underlying assumption is that this enables more refined decisions compared to crude policy choices such as discretized controllers , which limit the search space but induce abrupt changes . While switching controls can be undesirable in practice as they may challenge stability and accelerate system weardown , they are theoretically feasible and even arise as optimal strategies in some settings . It is therefore important to investigate our underlying assumption in designing policies for learning agents , and analyze how deviations from expected behavior can be explained . Practitioners have empirically observed that even under Gaussian policies extremal switching , or bang-bang control [ 31 ] , may naturally emerge ( e.g . in [ 21 , 41 , 51 ] ) . Thus , recent work focused on developing methods for preventing this behavior [ 7 ] or improving training when bang-bang control is the optimal policy structure [ 28 ] . However , understanding the performance , extent , and reasons for emergence of Bang-Bang policies in RL is largely an open research question . Its answer is important for designing future benchmarks and informing empirical and theoretical RL research directions . 1Correspondence to tseyde @ mit.edu . 2Equal advising . 3Please find videos and additional details at https : //sites.google.com/view/bang-bang-rl 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) In this paper we address these questions in two ways . First , we provide a theoretical intuition for bang-bang behavior in reinforcement learning . This is based on drawing connections to minimumtime problems where bang-bang control is often provably optimal . Second , we perform a set of experiments which optimize controllers via on-policy and off-policy learning as well as model-free and model-based state-of-the-art RL methods . Therein we compare the original algorithms with a slight modification where the Gaussian policy head is replaced with the Bernoulli distribution resulting in a bang-bang controller . In addition to theoretical justifications , our empirical results confirm emergence of bang-bang policies in standard continuous control benchmarks . Across the board , our experiments also demonstrate high performance of explicitly enforced bang-bang policies : the modified policy head can even outperform the original method . In connection to optimal control theory , we demonstrate how action costs empirically lead to sub-optimal performance with bang-bang controllers . However , we also show the negative impact of action penalties on exploration with Gaussian policies . Due to the necessity of exploration in RL to find the optimal solution , this can result in a complex trade-off . We also provide a discrete action space version of Maxmimum A Posteriori Policy Optimization ( MPO ) [ 1 ] that avoids relaxations or high variance gradient estimators , which many algorithms rely on when replacing Gaussians with discrete distributions . This is particularly useful as exploration , learning process and final performance are highly entangled in RL , and inaccurate or biased gradients can strongly affect learning . We furthermore aim to mitigate this entanglement by including results for distilling behaviour of a trained Gaussian agent into both a continuous and a discrete agent to compare their performance and provide further evidence for emergent bang-bang behavior . In summary , our work contains the following key contributions : 1 . We show competitive performance of bang-bang control on standard continuous control benchmarks by adapting several recent RL algorithms to leverage Bernoulli policies . 2 . We draw theoretical connections to optimal control , motivating the emergence of bang-bang behavior in certain problem formulations even under continuous policy parameterizations . 3 . We discuss the introduction of action penalties as a common method to reduce the emergence of bang-bang behaviour , and highlight resulting trade-offs with respect to exploration . 2 Related Work . Policy Representation Policies for continuous control are typically chosen to have continuous support and are commonly represented as Gaussians with diagonal covariance matrices [ 43 , 16 , 1 , 18 ] , Gaussian mixtures [ 56 ] , Beta distributions [ 11 ] , or latent variable models [ 15 ] . Exploration is then achieved by sampling from the continuous distributions with recent work relating maximum entropy objectives to the need for maintaining sufficient exploration in bounded action spaces [ 54 ] . Some approaches consider discretizing the action space to facilitate exploration [ 14 ] , but need to address the underlying curse of dimensionality [ 3 , 12 , 34 , 50 , 58 ] . The work by Tang and Agrawal [ 48 ] is the most closely related to our work showing that action space discretization can yield state-of-the-art performance for on-policy learning based on a sufficiently large number of discrete components . Here , we provide insight into how even the most extreme discretization of only bang-bang actions can yield state-of-the-art performance for a variety of on-policy and off-policy algorithms . Bang-Bang Policies Bang-bang control problems have been considered very early on in reinforcement learning [ 53 , 30 , 2 ] . These early works mainly consider problems known to require such a policy type . While it was often observed that bang-bang policies naturally emerge [ 21 , 41 , 51 ] most literature focused on avoiding this behavior [ 7 , 19 , 25 ] , while some leveraged it for specific applications [ 23 , 26 ] . In contrast , our work focuses on understanding its nature , particularly in scenarios that were assumed to require continuous actions for obtaining an optimal controller . Relation to Control Bang-bang controllers have been extensively investigated in optimal control research . Early works date to the 50s and 60s [ 45 , 4 ] when it was shown that bang-bang policies arise as optimal controllers in minimum time problems [ 32 , 31 ] . More recent work in bang-bang control focused on , e.g. , further analysis [ 8 , 55 ] , numerical computation [ 9 , 38 , 39 ] , and consideration of more specialized systems [ 10 , 29 , 35 , 57 ] . Another related line of work are switched dynamical systems where each mode corresponds to a bang-bang configuration . In these settings , one typically optimizes for switching times given a mode sequence [ 47 ] or penalizes the switching frequency as the control effort [ 46 ] . Unlike our work , research in this space typically does not consider the dynamics of the learning process in a deep RL setting . However , we can still leverage this existing body of work to inform our explanation for the seemingly surprising emergence of bang-bang policies in RL . 3 Optimal Control with Continuous-Time Deterministic Dynamics . We start by describing connections between the emergence of bang-bang behavior in optimal control and common continuous control reinforcement learning problems . First , we characterize three common reward structures together with the optimal controllers they induce . Then , we frame the solutions and their underlying assumptions in the context of our reinforcement learning setting . We consider a continuous-time deterministic dynamical system with state s ∈ Rn and action a ∈ R. The continuous time-setting simplifies analysis and provides a good approximation under the high sampling rates that are common on continuous control benchmarks and real-world robotic system . We define the non-linear control-affine system dynamics together with the objective function as ṡ ( t ) = f ( s ( t ) ) + g ( s ( t ) ) a ( t ) , 0 ≤ t ≤ T , maximize ∫ T 0 r ( s ( t ) ) − c ( a ( t ) ) dt , where f : S → S , g : S → S , t is the time , T is the final time , r : S → R is the state reward and c : A → R the action cost . We assume action a ( t ) to be a scalar in the set A = { a ( t ) ∈ R | |a ( t ) | ≤ 1 } . The following derivations can be directly extended to multidimensional inputs [ 27 , Sec . 5.2 ] . In optimal control settings , this is commonly solved by formulating the corresponding Hamiltonian H ( s ( t ) , a ( t ) , p ( t ) ) = r ( s ( t ) ) − c ( a ( t ) ) + p ( t ) T ( f ( s ( t ) ) + g ( s ( t ) ) a ( t ) ) , where p ( t ) is the costate variable , and leveraging Pontryagin ’ s maximum principle [ 6 , Prop . 3.3.1 ] [ 27 , Sec . 5.4 ] to derive the necessary optimality condition for action a ? ( t ) ∈ A and 0 ≤ t ≤ T as H ( s ? ( t ) , a ? ( t ) , p ? ( t ) ) ≥ H ( s ? ( t ) , a ( t ) , p ? ( t ) ) , ∀a ( t ) ∈ A . Maximum state reward ( MS ) Consider the case when c ( a ( t ) ) = 0 . Based on Pontryagin ’ s maximum principle , the optimal control is determined by solving the following simple linear program maximize p ? ( t ) T g ( s ? ( t ) ) a ( t ) subject to −1 ≤ a ( t ) ≤ 1 . Based on the optimization surface in Figure 1 ( left ) we see that the optimal action is given by a ? ( t ) = −1 p ? ( t ) T g ( s ? ( t ) ) < 0 undetermined p ? ( t ) T g ( s ? ( t ) ) = 0 1 p ? ( t ) T g ( s ? ( t ) ) > 0 . If the undetermined condition does not occur , this is referred to as bang-bang control . Minimum fuel-type cost ( MF ) Consider the case when c ( a ( t ) ) = |a ( t ) | . Through a slight reformulation under introduction of optimization variable z we arrive at another simple linear program maximize z + p ? ( t ) T g ( s ? ( t ) ) a ( t ) subject to z ≤ a ( t ) ≤ −z −1 ≤ a ( t ) ≤ 1 . Based on the optimization surface in Figure 1 ( right ) we see that the optimal action is given by a ? ( t ) = −1 p ? ( t ) T g ( s ? ( t ) ) < −1 0 |p ? ( t ) T g ( s ? ( t ) ) | < 1 1 p ? ( t ) T g ( s ? ( t ) ) > 1 undetermined |p ? ( t ) g ( s ? ( t ) ) | = 1 . If the undetermined condition does not occur , this yields bang-off-bang control ( see Appendix A ) . Minimum energy-type cost ( ME ) Consider the case when c ( a ( t ) ) = a ( t ) 2 . In general , this formulation leads to non bang-bang optimal control [ 6 , Example 3.3.2 ] . Singular arcs and chattering behavior In the undetermined conditions the Hamiltonian is no longer dependent on a ( t ) and the necessary condition does not allow for determining a ? ( t ) . These cases are commonly referred to as singular arcs . A sufficient condition to avoid singular arcs is having a linear time-invariant dynamical system that is controllable [ 27 , Sec . 5.6 ] . If an optimal control problem has singular arcs , a bang-bang solution may have to switch infinitely often in a finite time interval to achieve the desired state s ? ( t ) . This behavior is referred to as chattering [ 37 ] or Zeno ’ s phenomenon [ 59 ] . It does not appear in our formulation as we are considering discrete-time dynamical systems with a finite number of discretization points . Discretization effect The Pontryagin maximum principle can not be directly extended to discretetime dynamical systems without introducing additional assumptions . For example , Bertsekas [ 6 , Prop . 3.3.2 ] derives a discrete time version which , in order to provide necessary optimality conditions , requires convexity of the set A . If we consider only bang-bang actions , A is not convex and the discrete-time derivations no longer hold [ 6 , Example 3.10 ] . Nevertheless , we find that our discretization intervals are sufficiently small to approximate the continuous case ( see also Appendix C ) . Stochastic dynamics This simplified setup can be seen as a special case of reinforcement learning problems because of its deterministic dynamics . However , similar derivations can be done in case of stochastic action policies and dynamics . For example , this is the case when ṡ ( t ) = f ( s ( t ) ) + g ( s ( t ) ) a ( t ) + w ( t ) , where w ( t ) ∈ Rn is a random disturbance . Stochastic dynamics , together with the time discretization , are the key components to bridge this setup and the standard RL frameworks , which are discussed in the next section . Optimal controller Solving optimal control problems using the Pontryagin maximum principle is , in general , very challenging since it requires the numerical solution of a set of partial differential equations [ 6 , 27 ] . Instead , we directly learn switching controllers as stochastic Bernoulli policies .

Summary: The work investigates emergence of bang-bang control policies in continuous RL problems, by making the connection to optimal control. Exmpirically, it shows how continuous control problems are solved with bang-bang policies, using 4 different algorithms, and that for most of the domains in these experiments, a continuous action space is not needed. The experiments also explore modified reward functions designed to increase smootheness of the learnt policies and their effect on the (original) return obtained during exploitation.