Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies

1 Introduction . Real-world robotics tasks commonly manifest as control problems over continuous action spaces . When learning to act in such settings , control policies are typically represented as continuous probability distributions that cover all feasible control inputs - often Gaussians . The underlying assumption is that this enables more refined decisions compared to crude policy choices such as discretized controllers , which limit the search space but induce abrupt changes . While switching controls can be undesirable in practice as they may challenge stability and accelerate system weardown , they are theoretically feasible and even arise as optimal strategies in some settings . It is therefore important to investigate our underlying assumption in designing policies for learning agents , and analyze how deviations from expected behavior can be explained . Practitioners have empirically observed that even under Gaussian policies extremal switching , or bang-bang control [ 31 ] , may naturally emerge ( e.g . in [ 21 , 41 , 51 ] ) . Thus , recent work focused on developing methods for preventing this behavior [ 7 ] or improving training when bang-bang control is the optimal policy structure [ 28 ] . However , understanding the performance , extent , and reasons for emergence of Bang-Bang policies in RL is largely an open research question . Its answer is important for designing future benchmarks and informing empirical and theoretical RL research directions . 1Correspondence to tseyde @ mit.edu . 2Equal advising . 3Please find videos and additional details at https : //sites.google.com/view/bang-bang-rl 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) In this paper we address these questions in two ways . First , we provide a theoretical intuition for bang-bang behavior in reinforcement learning . This is based on drawing connections to minimumtime problems where bang-bang control is often provably optimal . Second , we perform a set of experiments which optimize controllers via on-policy and off-policy learning as well as model-free and model-based state-of-the-art RL methods . Therein we compare the original algorithms with a slight modification where the Gaussian policy head is replaced with the Bernoulli distribution resulting in a bang-bang controller . In addition to theoretical justifications , our empirical results confirm emergence of bang-bang policies in standard continuous control benchmarks . Across the board , our experiments also demonstrate high performance of explicitly enforced bang-bang policies : the modified policy head can even outperform the original method . In connection to optimal control theory , we demonstrate how action costs empirically lead to sub-optimal performance with bang-bang controllers . However , we also show the negative impact of action penalties on exploration with Gaussian policies . Due to the necessity of exploration in RL to find the optimal solution , this can result in a complex trade-off . We also provide a discrete action space version of Maxmimum A Posteriori Policy Optimization ( MPO ) [ 1 ] that avoids relaxations or high variance gradient estimators , which many algorithms rely on when replacing Gaussians with discrete distributions . This is particularly useful as exploration , learning process and final performance are highly entangled in RL , and inaccurate or biased gradients can strongly affect learning . We furthermore aim to mitigate this entanglement by including results for distilling behaviour of a trained Gaussian agent into both a continuous and a discrete agent to compare their performance and provide further evidence for emergent bang-bang behavior . In summary , our work contains the following key contributions : 1 . We show competitive performance of bang-bang control on standard continuous control benchmarks by adapting several recent RL algorithms to leverage Bernoulli policies . 2 . We draw theoretical connections to optimal control , motivating the emergence of bang-bang behavior in certain problem formulations even under continuous policy parameterizations . 3 . We discuss the introduction of action penalties as a common method to reduce the emergence of bang-bang behaviour , and highlight resulting trade-offs with respect to exploration . 2 Related Work . Policy Representation Policies for continuous control are typically chosen to have continuous support and are commonly represented as Gaussians with diagonal covariance matrices [ 43 , 16 , 1 , 18 ] , Gaussian mixtures [ 56 ] , Beta distributions [ 11 ] , or latent variable models [ 15 ] . Exploration is then achieved by sampling from the continuous distributions with recent work relating maximum entropy objectives to the need for maintaining sufficient exploration in bounded action spaces [ 54 ] . Some approaches consider discretizing the action space to facilitate exploration [ 14 ] , but need to address the underlying curse of dimensionality [ 3 , 12 , 34 , 50 , 58 ] . The work by Tang and Agrawal [ 48 ] is the most closely related to our work showing that action space discretization can yield state-of-the-art performance for on-policy learning based on a sufficiently large number of discrete components . Here , we provide insight into how even the most extreme discretization of only bang-bang actions can yield state-of-the-art performance for a variety of on-policy and off-policy algorithms . Bang-Bang Policies Bang-bang control problems have been considered very early on in reinforcement learning [ 53 , 30 , 2 ] . These early works mainly consider problems known to require such a policy type . While it was often observed that bang-bang policies naturally emerge [ 21 , 41 , 51 ] most literature focused on avoiding this behavior [ 7 , 19 , 25 ] , while some leveraged it for specific applications [ 23 , 26 ] . In contrast , our work focuses on understanding its nature , particularly in scenarios that were assumed to require continuous actions for obtaining an optimal controller . Relation to Control Bang-bang controllers have been extensively investigated in optimal control research . Early works date to the 50s and 60s [ 45 , 4 ] when it was shown that bang-bang policies arise as optimal controllers in minimum time problems [ 32 , 31 ] . More recent work in bang-bang control focused on , e.g. , further analysis [ 8 , 55 ] , numerical computation [ 9 , 38 , 39 ] , and consideration of more specialized systems [ 10 , 29 , 35 , 57 ] . Another related line of work are switched dynamical systems where each mode corresponds to a bang-bang configuration . In these settings , one typically optimizes for switching times given a mode sequence [ 47 ] or penalizes the switching frequency as the control effort [ 46 ] . Unlike our work , research in this space typically does not consider the dynamics of the learning process in a deep RL setting . However , we can still leverage this existing body of work to inform our explanation for the seemingly surprising emergence of bang-bang policies in RL . 3 Optimal Control with Continuous-Time Deterministic Dynamics . We start by describing connections between the emergence of bang-bang behavior in optimal control and common continuous control reinforcement learning problems . First , we characterize three common reward structures together with the optimal controllers they induce . Then , we frame the solutions and their underlying assumptions in the context of our reinforcement learning setting . We consider a continuous-time deterministic dynamical system with state s ∈ Rn and action a ∈ R. The continuous time-setting simplifies analysis and provides a good approximation under the high sampling rates that are common on continuous control benchmarks and real-world robotic system . We define the non-linear control-affine system dynamics together with the objective function as ṡ ( t ) = f ( s ( t ) ) + g ( s ( t ) ) a ( t ) , 0 ≤ t ≤ T , maximize ∫ T 0 r ( s ( t ) ) − c ( a ( t ) ) dt , where f : S → S , g : S → S , t is the time , T is the final time , r : S → R is the state reward and c : A → R the action cost . We assume action a ( t ) to be a scalar in the set A = { a ( t ) ∈ R | |a ( t ) | ≤ 1 } . The following derivations can be directly extended to multidimensional inputs [ 27 , Sec . 5.2 ] . In optimal control settings , this is commonly solved by formulating the corresponding Hamiltonian H ( s ( t ) , a ( t ) , p ( t ) ) = r ( s ( t ) ) − c ( a ( t ) ) + p ( t ) T ( f ( s ( t ) ) + g ( s ( t ) ) a ( t ) ) , where p ( t ) is the costate variable , and leveraging Pontryagin ’ s maximum principle [ 6 , Prop . 3.3.1 ] [ 27 , Sec . 5.4 ] to derive the necessary optimality condition for action a ? ( t ) ∈ A and 0 ≤ t ≤ T as H ( s ? ( t ) , a ? ( t ) , p ? ( t ) ) ≥ H ( s ? ( t ) , a ( t ) , p ? ( t ) ) , ∀a ( t ) ∈ A . Maximum state reward ( MS ) Consider the case when c ( a ( t ) ) = 0 . Based on Pontryagin ’ s maximum principle , the optimal control is determined by solving the following simple linear program maximize p ? ( t ) T g ( s ? ( t ) ) a ( t ) subject to −1 ≤ a ( t ) ≤ 1 . Based on the optimization surface in Figure 1 ( left ) we see that the optimal action is given by a ? ( t ) = −1 p ? ( t ) T g ( s ? ( t ) ) < 0 undetermined p ? ( t ) T g ( s ? ( t ) ) = 0 1 p ? ( t ) T g ( s ? ( t ) ) > 0 . If the undetermined condition does not occur , this is referred to as bang-bang control . Minimum fuel-type cost ( MF ) Consider the case when c ( a ( t ) ) = |a ( t ) | . Through a slight reformulation under introduction of optimization variable z we arrive at another simple linear program maximize z + p ? ( t ) T g ( s ? ( t ) ) a ( t ) subject to z ≤ a ( t ) ≤ −z −1 ≤ a ( t ) ≤ 1 . Based on the optimization surface in Figure 1 ( right ) we see that the optimal action is given by a ? ( t ) = −1 p ? ( t ) T g ( s ? ( t ) ) < −1 0 |p ? ( t ) T g ( s ? ( t ) ) | < 1 1 p ? ( t ) T g ( s ? ( t ) ) > 1 undetermined |p ? ( t ) g ( s ? ( t ) ) | = 1 . If the undetermined condition does not occur , this yields bang-off-bang control ( see Appendix A ) . Minimum energy-type cost ( ME ) Consider the case when c ( a ( t ) ) = a ( t ) 2 . In general , this formulation leads to non bang-bang optimal control [ 6 , Example 3.3.2 ] . Singular arcs and chattering behavior In the undetermined conditions the Hamiltonian is no longer dependent on a ( t ) and the necessary condition does not allow for determining a ? ( t ) . These cases are commonly referred to as singular arcs . A sufficient condition to avoid singular arcs is having a linear time-invariant dynamical system that is controllable [ 27 , Sec . 5.6 ] . If an optimal control problem has singular arcs , a bang-bang solution may have to switch infinitely often in a finite time interval to achieve the desired state s ? ( t ) . This behavior is referred to as chattering [ 37 ] or Zeno ’ s phenomenon [ 59 ] . It does not appear in our formulation as we are considering discrete-time dynamical systems with a finite number of discretization points . Discretization effect The Pontryagin maximum principle can not be directly extended to discretetime dynamical systems without introducing additional assumptions . For example , Bertsekas [ 6 , Prop . 3.3.2 ] derives a discrete time version which , in order to provide necessary optimality conditions , requires convexity of the set A . If we consider only bang-bang actions , A is not convex and the discrete-time derivations no longer hold [ 6 , Example 3.10 ] . Nevertheless , we find that our discretization intervals are sufficiently small to approximate the continuous case ( see also Appendix C ) . Stochastic dynamics This simplified setup can be seen as a special case of reinforcement learning problems because of its deterministic dynamics . However , similar derivations can be done in case of stochastic action policies and dynamics . For example , this is the case when ṡ ( t ) = f ( s ( t ) ) + g ( s ( t ) ) a ( t ) + w ( t ) , where w ( t ) ∈ Rn is a random disturbance . Stochastic dynamics , together with the time discretization , are the key components to bridge this setup and the standard RL frameworks , which are discussed in the next section . Optimal controller Solving optimal control problems using the Pontryagin maximum principle is , in general , very challenging since it requires the numerical solution of a set of partial differential equations [ 6 , 27 ] . Instead , we directly learn switching controllers as stochastic Bernoulli policies .

Summary: The paper focuses on the so-called bang-bang policies. These are ones, which are allowed only for discrete values, typically two extremal ones. Even though they are usually disadvantageous in real-world scenarios, they tend to arise in simulation, and they seem to deliver at least decent performance.