EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
Authors: Samuel Holt
Todor Davchev
Dhruva Tirumala
Ben Moran
Atil Iscen
Antoine Laurens
Yixin Lin
Erik Frey
Markus Wulfmeier
Francesco Romano
Nicolas Heess
Venue: ICML 2025 poster (2025)
Keywords: Hierarchical Reinforcement Learning
Evolutionary Strategies
High-Frequency Control
Continuous Control
Robotics
Primary area: reinforcement_learning->everything_else
TL;DR: EvoControl marries a slow PPO high-level policy with an evolution-strategies low-level controller to learn fast, adaptive, tune-free high-frequency continuous control that outperforms both PD-based hierarchies and direct torque baselines.

Abstract: High-frequency control in continuous action and state spaces is essential for practical applications in the physical world. Directly applying end-to-end reinforcement learning to high-frequency control tasks struggles with assigning credit to actions across long temporal horizons, compounded by the difficulty of efficient exploration. The alternative, learning low-frequency policies that guide higher-frequency controllers (e.g., proportional-derivative (PD) controllers), can result in a limited total expressiveness of the combined control system, hindering overall performance. We introduce *EvoControl*, a novel bi-level policy learning framework for learning both a slow high-level policy (using PPO) and a fast low-level policy (using Evolution Strategies) for solving continuous control tasks. Learning with Evolution Strategies for the lower-policy allows robust learning for long horizons that crucially arise when operating at higher frequencies. This enables *EvoControl* to learn to control interactions at a high frequency, benefitting from more efficient exploration and credit assignment than direct high-frequency torque control without the need to hand-tune PD parameters. We empirically demonstrate that *EvoControl* can achieve a higher evaluation reward for continuous-control tasks compared to existing approaches, specifically excelling in tasks where high-frequency control is needed, such as those requiring safety-critical fast reactions.

EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
Samuel Holt 1 Todor Davchey? Dhruva Tirumala
Ben Moran 2 Atil Iscen Antoine Laurens? Yixin Lin
Erik Frey? Markus Wulfmeier =2 Francesco Romano :2 Nicolas Heess?
Abstract
1. Introduction
figh-frequency control is paramount for ensuring the safety and reliability of real-world robotic systems (Hogan, 1984; Oh et al., 2014; Venkataraman & Gulati, 1993; Dantec et al., 2022). Failures to respond in real-time to unexpected colli- sions, disturbances, or human interactions can lead to catas- trophic consequences in safety-critical applications such as surgery (Kuchenbecker al., 2010), autonomous driv- ing (Guo al., 2019), and industrial automation (Vasic & Billard, 2013). To achieve reliable performance, practi- cal robotic systems commonly rely on low-level fixed con- trollers (e.g., proportional-deriiatiee. (PD) controllers) oper- ating at high frequencies. However, this approach raises the fundamental question of whether we can remove these lower- level fixed controllers entirely and instead directly control motor torques high frequency, potentially enabling faster reactions and finer control.
High-frequency control in continuous action and state spaces is essential for practical applications in the physical world. Directly applying end-to- end reinforcement learning to figh-frequency con- trol tasks struggles with assigning credit to actions across long temporal horizons, compounded by the difficulty of efficient exploration. The alter- native, learning low-frequee frequerreccyy policies that guide higher-frequency controllers (e.g., proportional- derivative (PD) controllers), can result in a lim- ited total expressiveness of the combined control system, hindering overall performance. We intro- duce EvoControl, a novel bi-level policy learning framework for learning both slow high-level policy (using PPO) and fast low-level policy (us- ing Evolution Strategies) for solving continuous control tasks. Learning with Evolution Strate- gies for the lower-policy allows robust learning for long horizons that crucially arise when op- erating at higher frequencies. This enables Evo- Control to learn to control interactions a high frequency, benefitting from more efficient explo- ration and credit assignment than direct high- frequency torque control without the need to hand- tune PD parameters. We empirically demonstrate that EvoControl can achieve a higher evaluation reward for continuous-control tasks compared to existing approaches, specifically excelling tasks where figgueeey control is needed, such as those requiring safety-critical fast reactions.
Recent work falls into two main categories. First, directly learning end-to-end high-frequency (often torque-based (Al- jalbout et al., 2024)) policies, labeled direct torque control, presents significant learning challenges as this increases the number of state transitions within fixed time window, resulting in longer trajectories with complex temporal de- pendencies that hinder exploration and credit assignment (Peng & Van De Panne, 2017; Sutton & Barto, 2018; Dab- ney et al., 2020) and can often lead to suboptimal behavior (Section 5.2).
Second, common alternative to learn a slower high-level policy that outputs target positions or velocities, which are then tracked by faster (higher- frequecc) low-level fixed controller, such as a PD controller (Wei, 2019). This bi- level approach, labeled as fixed controllers, is prevalent in real-worid continuous control tasks ( (Aljalbout et al., 2024; Song et al., 2019) and simplifies learning by reducing the effective time horizon of the high-level policy (Sutton et al., 1999). For real-time control, this composition affords the high-level policy larger inference time, key for handling rich observations, such images (i.e., 30Hz) or using larger networks, such as Visual-Language-MM 2024). While composing with a low-level PD controller that operates at frequenies ~ 500H on robotic plat-
*Work done during an internship Google DeepMind. *Equal advising. 'University of Cambridge *Google DeepMind. Corre- spondence to: Samuel Holt <sih31 @cam.ac.uk>.
(Ma et al.,
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

EvoControl: Multi-Frequeng, Bi-Level Control for High-Freqqenny Continuous Control
forms, often only observing high-frequency direct robot proprioceptive observations such robot joint positions, velocities and torques (Borase et al., 2021). However, fixed controllers are unable to produce fast interaction behavior beyond simple satt-goal-tracking limiting their expressive- ness, and require manual tuning of their PD parameters for each task.
2. Problem
We follow the standard continuous control reinforcement learning (RL) setting with the inclusion of an optional low- level controller.
States & Actions. We denote the environments state space as S C Rd. and its action space as U C IRd At time E R, the system's 's state is represented by S E S, and its action by u, E U. Considering action (e.g.. actuator) limits, the action space is constrained to a box in Euclidean space:
An effective method for high-frreuency control therefore aims to have the following three core properties:
(P1) Efficient Exploration: Throughout learning, be able to efficiently explore the state space as well as high-level policy with a fixed controller.
U mmin: mmax]
Environment Dynamics. The transition dynamics for con- tinuous control environments can be described by an un- derlying unknown differential equation of 6 da- (s,, ut). The transition function, which describes the evolution of the state over a discrete time step Ås, can be approximated using the Euler method 8t40, Ri 8, + (ef(s, uz). Given an action ue and current state St, St+A, ~ P(s++ A, 18t, Ut) is implicitly defined by this ap- proximation. More sophisticated numerical integration methods can also be used. We expand on the problem setup in Appendix A.
Enable the learning of a low-level controller capable of complex, adap- tive behaviors at high frequencies.
(P2) High-Frequency Interaction Control:
dt
(P3) Automate Controller Tuning: Reduce manual tuning of the low-level PD controller parameters.
With these considerations, we introduce EvoControl, a novel bi-level policy learning framework for learning both a slow high-level policy and a fast low-level policy for continuous control robotic tasks. EvoControl learns a high-level policy with proximal policy optimization (PPO) and low-level proprioceptive policy with Evolution Strategies (ES) in alter- nating stages. The low-level policy is initialized as a fixed PD controller and gradually transitions to a learned neural network policy throughout training via an annealing param- eter a. This staged training process enables stable learning and effective exploration. ES at the low level provides ro- bust learning, particularly beneficial for long horizon credit assignment inherent in high-frequency control. These com- ponents enable EvoControl to learn high-frequency interac- tion control, automate controller tuning, and more efficiently explore than direct high-freque high-frequency torque control.
The agent can be represented as a single policy т. Rdd + Rdd, that observes the current observation at time and samples an action UN (T(s) and then applies this action to the environment a given fixed A, To formalize
Policies.
Algorithm Bi-Level Policy Interaction (Single High-Level Step)
I: On ~ p(*)) (High-level action]
2: for 0
do
uk+ ~ ((K++s ax) (Low-level action)
4: Sk+i+1 ~ f (8k+s
Contributions: I We introduce EvoControl, a novel bi- level policy leamning framework for learning both a slow (e.g. 30Hz) high-level policy and a fast (c.g. 500Hz) low-level robot proprioceptive controller using PPO and ES, respec- tively, for continuous-control tasks (Section 3). ② Theo- retically, we show that there exist some continuous-tima Markov Decision Processes (CTMDPs) in which acting at higher frequencies can yield strictly higher expected cumula- tive reward (Proposition 2.1). Empirically, we demonstrate that EvoControl can achieve a higher evaluation reward for standard continuous control tasks at high frequency com- pared to existing approaches, excelling in tasks where high- frequency control is needed, such as in safety-critical appli- cations of unmodeled interactions (Section 1. ④ We gain insight and understanding of how EvoControl can achieve efficient exploration compared to direct torque control at high-frequency, learn fast interactions, and demonstrate ro- bustness to mistuned PD parameter settings.
5: end for
bi-level policy, we decompose ' into two components: slow, high-level policy, p, and a fast, low-level policy, B. Both policies interact with the environment as described in Algorithm 1. At time step k. outputs latent action ag ~ p(sx) (e.g., a target position or velocity). Operating at a higher frequency, /3 receives ag and generates low-level actions uk+i (c.g., motor torques) at a finer-grained time step to achieve the target specified by p. With 3 operating at frequency 1/ At, p's latent action is exccuted by /3 for G steps, making p's effective frequency 1 1/(GAt).
Objective. The environment produces reward r. sampled from an unknown reward function r(st, uy). The overall objective of the agent to maximize the expected future dis- counted reward E so.r: 1.07-1.707-1
D==0 //r] where
0 5 < 1 is the discount factor.
2

---

EvoControl: Multi-Frequency Bi-Level Control for High- High-Frequency Continuous Control
2.1. Existence of CTMDPs Requiring Higher-Frequenny Control for Optimality
Table I: Common Fixed Low-Level PD Controllers
Method
Control Law
In certain continuous- time Markov Decision Processes (CT- MDPs), taking actions at higher frequency strictly in- creases the achievable reward-a a point we make formally in Proposition 2.1.
PD Absolate Position
r(t) K,ig"
Kals"
PD Deha Positioe
66*
r() K, 6og) - 9) Kil i)
PD Velocity
8 0.
((1 - K,10" 4) KG 4)
PD Integrated Velocity
(()) "ollo" + Fatat - + Kat0 4)
PD Position K2
40% (p) m
x00 Kgl0" 4) Kali"
Proposition 2.1. Consider a continuous-fime Markov De- cision Process (CTMDP) with a finite state space and action space. Let this CTMDP be discretized with time step D.- There exist CTMDPs such that for any fixed dis- cretization step Â > O, there exists a finer discretization A/ < At where the optimal policy for the Af-discretized MDP achieves a higher expected cumulative reward over a fixed time horizon T than the optimal policy for the D discretized MDP.
often desired position or velocity', which the low-level PD controller then tracks. The controller computes a control signal ut based on the error between the target ag and the measured system state 81, We = Kp((x - 8.) + Ka(āx - 8t) where e, = aj. -8 St is the tracking error, and Kp. Ka E are constant proportional and derivative gains. Common PD controller designs using proprioceptive states (joint po- sitions yt. velocities ét, and torques T₁) are summarized in Table 1. We provide an expanded background on PD controllers in Appendix C.
Proof. Full proof in Appendix B; however, we present the following sketch. Consider CTMDP with states {Sgond, Sbud) } and actions {Ammiimain Orecover). }. Under Ommimain- the system transitions from 5good to Shad at rate p > 0, while Drecover immediately transitions Sbad to Sgood. Let rewards be r(8good) = Y good > O and r' r(Abad) = !bad < 0, with episode length T. Discretizing at As, the optimal policy uses Omaintain when in 8yoad and (recovver when in Shad, yielding expected return R(D,) = T Tgoo - T p (Tgood Tbad). Now refine to A De/n where (n > 1). Recovery hap- pens more quickly, reducing time spent in Sbad each A,- interval, so the finer-discretized return is R ({)))= = T "good Tp (Tgood /ba) The difference R(A,) R(A,)
3. EvoControl: Evolved Low-Level Controller Framework
We now propose EvoControl, a novel bi-level policy learn- ing framework for learning both a slow high-level policy and fast low-level policy for continuous control tasks. The key idea is to stabilize the bi-level on-policy learning of higher-level policy by initially learning with fixed- low-level PD controller and then annealing to gradually le-lendeed high- figh-frequency controller- 1 provides a block diagram.
n T As(Tgood - rbwa)(1 (1- 2 O, so a finer discretization leads to a strictly higher expected cumulative reward under D
First, we formulate the bi-level policy optimization problem and its challenges. Then, we discuss our approach and the advantages of using ES for lower-level policy optimization. Specifically our framework can be applied starting with dif- ferent semantically meaningful high-level actions ag from the high-level policy, such as resition/velocity targets, com- monly seen in existing PD controllers, as outlined in Table 1.
its optimal policy.
This highlights the need for figher-rrequecy control in cer- tain environments. This is analogous to the Pulse Width Modulation (PWM) sampling theorem (Huang et al., 201 1), where variable pulse widths enable perfect signal reconstruc- tion from discrete samples, similar to how actions enable optimal control in our MDP setting. We further explore a continuous control safety-critical intuitive example in Appendix B.
3.1. Promise and Challenges of Policy Hierarchies
Hierarchical reinforcement learning (HRL), employs multi- ple levels of policies with increasing temporal abstraction. It tackles complex tasks through improved exploration and long-horizon planning (Parr & Russell, 1997; Sutton et al., 1999; 2011). If high-level actions induce diverse low-level trajectories, exploration can become more effective (Sutton et al., 1999; Li et al., 2021; McClinton et al., 2021). For instance, fixed low-level PD controllers, combined with a learned high-level policy, act as temporally extended ac- tions, facilitating efficient exploration if aligned with the
2.2. Background: Fixed PD Controllers
In continuous control and robotics, hierarchical structures composed of a learned high-level policy (p) and fixed low- level controller (e.g., a PD controller (Oku et al., 2018; Find- eisen et al., 1980)) are common. This hierarchical decompo- sition reduces the number of decision steps for the high-level policy by a factor of G within a fixed episode duration T, where G is the number of low-level actions exccuted per high-level action. The high-level policy outputs a target (k,,
'The state 8t can encompass wide range of proprioceptive information beyond joint positions (qt). We present the target as and tracking error in terms of position/velocity align with common PD controller formulations.
3

---

EyoControl: Multi- Bi-Level Control for High-Frequency Continuous Control
Bi-Level Rollout - Following A Algorithm
High-Level Policy Po(
Po (sG)
Po(su)
High-Level do
AG +
Action
Low-Level Policy (so, 0o)
Ba(51, (o)
8,(80-1,00)
Ba (a ag)
Ba Ba(#G+1,0c) ac)
Ba(82G-
Ao(s)ts) 9x)
?
2
2
2
2
M1
UG-1
UG
UG+1
W2G
+(1 - a) ()
81
SG+1
SG+2
82G
B) EvoControl Training Loop
Evolution Strategies
for in (0....KK K}:
PPO(8.6.0) 0.0) Update 9, Fix
Sample
T/A.-1
Evaluate
Update
FS(0....)
Fix 8, Update
Øi
F(d:) =
Σ
r(sia, UiA, ) Φ Φ
k/K)
Repeat
O
gencrations
Figure 1: A) Bi-level Policy Interaction. The high-level policy Pa outputs latent action ag, which guides the low-level policy Sg for G steps. B) EvoControl Training Loop. EvoControl trains both the high-level (pe. with parameters 0) and low-level (Be, with parameters () policies over the course of training divided into K discrete sections. Each section l: first optimizes with PPO (fixed Φ), then optimizes & with Evolution Strategies (fixed 0). ES maintains a population d parameters, evaluates their fitness (episodic return, F), and updates the parameter distribution to maximize average fitness. This process, robust to long horizons, enables learning of complex low-level behaviors. To stabilize learning, Bg is initially PD controller (3pp, 1) and transitions to a learned controller anneals towards O.
task2 (Chiaverini et al., 1999).
3.2. Efficient High-Level Policy Exploration
However, simultaneously learning high- and low-level poli- cies with an intermediate latent action can destabilize learn- ing (Yang et al., 2021; Nachum al., 2018a; Wöhlke al., 2021). As the low-level policy updates, the same high- level latent action may produce different low-level action sequences, creating non-stationary learning environment for the high-level policy. Conversely, effective low-level pol- icy learning requires informative high-level latent actions, creating co-dependency that can hinder both levels of the hierarehy.
In the following, we outline how we train both the slow high- level policy / and the fast low-level policy / by training each level in K E Σ44 stages, with the other fixed, which assists in mitigating the issues of instability and long-horizon credit assignment.
To deliver on the key advantage of efficient high-level state- action exploration for the bi-level control approach, as out- lined in Figure 1, we seek stable way of initially learning the high-level policy to overcome the non-stationary chal- lenge of learning with a continually updating lower-level policy. A key approach we choose, which, as we will see later, also assists in stably learning lower-level policy, is to represent the lower-level policy as convex combination of fixed PD controller 3pp and learned neural network actor policy SONN with parameters ø, and start initially training with only the PD controller, with a = 1, E [0, 1] and anneal a to 0 over the course of training. Specifically, we formulate the bi-level policies during training Algorithm I as:
Bi-level learning presents further challenges: determining optimal latent actions (Nachum et al., 20) and the appro- priate low-level policy reward function to optimize. While subgoal-based rewards (e.g., = -I|šgoal promising and relate to PD control. they can struggle to capture complex behaviors best optimized through overall episodic return (R = ETAA r(si,, U,d,) Further- more optimizing R requires long-term credit assignment, exacerbated by high-freque fighuerey low-level policies (Peng & Van De Panne, 2017). This motivates our framework for optimizing the episodic return R directly.
sel12) - are
(o(s))
Ba ( ax) = (S ax) + (1 BANN ax)
FFrr example, a PD absolute position controller can serve as an effective prior for goal-reaching tasks, where the low-level control actions us correspond to torques applied to the robot's actuators.
This brings two immediate advantages: 1) the high-level action ds output from the high-level policy has an initial semantic meaning, grounding it and allowing a user to select the most appropriate PD controller for the given task, and 2)

---

EvoControl: Multi-Frequency Bi-Level Control for High-Freqqenny Continuous Control
as a is annealed throughout training from 1 to O, we inherit the effective state-action exploration properties of having fixed PD controller initially (Section 5.2), and yet can still retain the flexibility of learning more complex lower-level behavior, beyond just sub-goal/state tracking.
networks) (Rechenberg, 1973; Wierstra et al., 2014; Sali- mans et 2017). ES is a gradient-free, black box, global optimization method that optimizes the lower-level policy neural network parameters o. by maintaining a population of parameters represented by a distribution pr((), itself pa- rameterized by 17 and maximizes the average fitness value Ego-,, F(Ø) over the population by searching for e with stochastic gradient ascent (Salimans et al., 2017). The core idea rests upon optimizing the score function estimator of
To optimize the high-level policy, we employ Proximal Pol- icy Optimization (PPO) (Schulman et al., 2017), a highly effective on-policy reinforcement learning algorithm for continuous control tasks. The high-level policy is repre- sented by continuous control agent consisting of neural network with separate critic and actor heads. The actor head parameterizes a multivariate Gaussian distribution with a diagonal covariance matrix, with parameters A. Specifically, the high-level policy is defined as:
1 -E + ge)e],
V VoE.NN(O.,)1F + oe)] =
where (V, is the standard multivariate normal distribu- tion, and a is a step size parameter. This estimator allows for gradient estimation without explicit backpropagation by sampling perturbations of c. Therefore, ES maintains a pop- ulation of parameters, evaluates their fitness, and generates a new population through selection, mutation (adding noise oc), and recombination, as determined by specific ES al- gorithm used (Salimans et al., 2017). ES, while less sample efficient compared to RL, are particularly well-suited for scenarios where gradient-based methods struggle, such as those with delayed rewards, noisy environments, or long- horizon tasks (Salimans et al., 2017). The EvoControl frame- work supports different ES algorithms, and we empirically evaluate many competitive ES approaches in Appendix J.1 and find that the competitive approach of Policy Gradients with Parameter- Based Exploration (PGPE) (Sehnke et al., 2010) is both effective within EvoControl (Section 5.1) and straightforward to implement.
ak* ~ po(sk) = Eo(ss))
where (ss) is the mean vector and Eo(sk) diag(o), (Sk), (2)) "a....,,,n (sk)) is the diagonal co- variance matrix, with (sx) representing the variance for the 1-th action dimension at time step k.
3.3. ES-learning a Fast Low-level Policy
We seek to learn more complex lower-level behavior be- yond prior work of simple goal-reaching low-level policies (Nachum et al., 201 8a). However, directly optimizing the episodic return (/?) is challenging due to the extended credit assignment horizon, exacerbated by the low-level controller and its increased number of steps (G)
per high-level latent action ag.
One consideration is what to select as the fitness function F()) for the parameters. Given the bi-level setup, we seek to optimize the episodic return for the combined bi-level pol- icy as a rollout in the environment, i.c. F(ò) R. Doing so becomes a long horizon optimization problem, especially when the lower-level operates at higher frequency. Inter- estingly, directly optimizing the parameters with gradient descent is infeasible due to stochastic noise on the state of the environment and potentially many steps of gradient propagation through entire li-le policy rollout of the environment. Crucially, to reduce the variance of the fit- ness function F and improve learning convergence of the lower-level policy, we sample the mode of the probabilistic high-level actor, and parameterize the lower-level actor as a deterministic low-level policy neural network.
Policy gradient methods, while a natural choice for pol- icy optimization, are known to struggle with the long hori- zons encountered in high-freuenny control (Peng & Van De Panne, 2017)². This difficulty is compounded by the credit assignment problem (Sutton & Barto, 2018), where the impact of individual low-level actions on the overall return becomes increasingly diffused over longer horizons (Dayan & Hinton, 1992). Furthermore, the increased tem- poral density of actions at higher frrqueeciess can lead to heightened sensitivity to policy parameter variations, mak- ing the optimization landscapes more challenging to navi- gate and potentially leading to suboptimal solutions. Em- pirical evidence supporting these challenges is presented in Appendix J.2.
Instead, we seek learning method that is invariant to the long-horizon credit assignment issue and that can dis- cover globally optimal lowelevel policy. Motivated by this, we adopt an Evolutionary Strategies (ES) approach (of- ten termed Neuroevolution when applied to training neural
Another advantage of using ES for lower-level policy opti- mization is the inherent parallelism of fitness evaluations. This parallelism can lead to faster wall-clock time conver- gence compared to gradient-basee baaed methods, even though ES generally requires more samples (Salimans et al., 2017). In our experiments, we find that this trade-off is benefi- cial: the increased sample complexity is outweighed by the ability to stably learn a high- fieeueeyy low-level policy
Low-level controllers often operate at 500H>-1KHz, resulting in G 50- 100 low-level actions respectively for each high- level action at 10Hz.
5

---

EvoControl: Multi- Hrequ--rre Bi-Level Control for High-Frequeng, Continuous Control
(Appendix J.3).
Table 2: EvoControl Ablation of PD Controllers.
The process of annealing with a PD controller further im- proves learning a lower-level policy that can be directed with a high-level latent action as input ag that is directed towards solving the task. Although initially the higher-level policy will provide a latent-action ag that would be applica- ble a particular PD controller that it was initially trained with in the early stages of training, by enabling full opti- mization of the lower-level policy we can evolve a better performing lower-level policy, lessening the reliance on tuned PD controller. In practice we find this effective, even the PD) controller is mistuned (Section 5.2). We provide pseudocode for EvoControl in Appendix G.I.
Controller Variant
SooN Obs.
Bas action
EvoControl (Full Stute)
Rei Ray Cay Res fit, t/T
EvoControl (Residual State)
EvoControl (Target + Proprioceptive)
Me fit, R1, L/T
7
EvoControl (Target)
dee Mh dit T/T
EveControl (Learned Gains)
"as Me Whe the L/T Re, agi Ray Res t/T
Kg Ka
EveControl (Delta Position)
T
Hierarchical Reinforcement Learning (HRL): Methods, including options frameworks (Sutton et al., 1999; Bacon et al., 2017), hierarchical actor-critic architectures (Ried- miller et al., 2018; Vezzani et al., 2022), and recent exten- sions to effectively train deep policies and critics (Rao et al., 2021; Salter et al., 2022; Wulfmeier et al., 2020a;b), decom- pose complex tasks into simpler subtasks. While these meth- ods have demonstrated success in improving exploration and learning, they typically focus on discrete skill/subgoal decomposition. Furthermore, HRL subgoal methods of- ten learn simpler lower-level policies limited by subgoal attainment, rather than optimizing overall episode return for complex behavior. EvoControl, inspired by HRL, ad- dresses the challenges of continuous high-frequency control with semantically meaningful exploration. Unlike typical HRI., which focuses on skill discovery, EvoControl targets learning a fast low-level policy that complements the slow high-level policy. Uniquely, EvoControl combines PPO and ES within its hierarchical framework for efficient explo- ration and complex high-frequency control.
4. Related Work
Here provide the existing approaches to continuous con- trol, and provide an extended related work in Appendix D.
Fixed Low-Level Controllers: Commonly involve com- bining a Icarned high-level policy (often operating at low frequency, e.g., 10-30Hz) with a fixed or analytical high-frequency low-level controller (e.g.. a Proportional- Derivative (PD) controller operating at 500Hz higher) (Song et al., 2019). Whilst prevalent, due to the ease of state-action space exploration (Peng& & Van De Panne, 2017; Pateria et al., 2021), this approach suffers from several draw- backs. The low-level controller's PD parameters require careful tuning per task, and its fixed nature limits its ability to handle high-frequency interactions, such as unexpected collisions disturbances, that involve more complex be- havior than just reaching a given goal state or emergency braking. Furthermore, recent work applying RL algorithms to the physical world often restricts itself to relatively low- frequency control (~~20Hz) due to the reliance on analytical impedance controllers (Martín-Marttin et al., 2019; Luo et al., 2018; Johannink al., 2019; Davehev al., 2022). Evo- Control, in contrast, aims achieve efficient exploration while also enabling the learning of flexible and complex high-frequency behaviors in the low-level policy.
5. Experiments and Evaluation
In this section, we evaluate EvoControl and verify that it can achieve a higher evaluation reward for both the same number of high-level policy steps and the same number of low-level enviormment steps compared to the existing training of a high-level policy either with fixed controllers or direct torque control.
Benchmark Environments. We evaluate performance on thirteen digh-dimensional continuous control environments. Ten environments are adapted from standard Gym MuloCo tasks (Brockman et al., 2016a; Freeman al., 2021), in- cluding locomotion (e.g., Ant, HalfCheetah, Humanoid) and manipulation tasks (c.g.. Reacher, Pusher). Crucially, we substantially modify these benchmarks by increasing the control frequency to 500Hz (with episodes lasting 1000 steps or 2 seconds of real-time) and removing the typical control-cost term. Typical Gym MuJoCo tasks operate at control frequencies between 1205-100Hz, whereas mod- ifications explicitly examine whether directly controlling motor torques at significantly higher frequencies can offer advantages such as faster reactions and finer motor con- trol. Additionally, we introduce two novel safety-critice variants Reacher and HalfCheetah with randomly posi-
Direct Torque Control: Methods learn end-to-end policies that output joint torques at high frequency (Peng & Van De Panne, 2017: Wahlström et al., 2015; Watter et al., 2015). While offering greater flexibility, this approach suffers from the curse of dimensionality imposed by the increased num- ber of time steps in long horizons. The resulting explosion in the number of possible action sequences significantly hinders exploration and can lead to suboptimal policies (Martín-Je -arrfnn-Mattnnn et al., 2019; Peng & Van De Panne, 2017). EvoControl mitigates this challenge by employing a hierar- chical structure, enabling more efficient exploration while retaining the adaptability afforded by direct torque control at the low level.
6

---

EvoControl: Multi-Frequeee Bi-Level Control for High-FFeqqenny Continuous Control
Table 3: Normalized evaluation returns (R) for benchmarks trained for an equivalent number of IM high-level (p) steps per environment. EvoControl consistently outperforms baseline methods (fixed controllers and direct torque control), with results averaged over 384 random seeds (95% confidence intervals shown). Scores are normalized between 0 (random policy) and 100 (best-performing con-EvoConttro baseline). Bold values exceed the best baseline performance (scores >000). Corresponding unnormalized results are provided in Table 28.
Same PPO high level dlg, with LowweLeve Pollcy 2
Halkchetthh X
Hopper X
Hummaodd X-1
Humanoid Stundup
Inverted Double Fend.
Inverted Pend.
Puther x11
Reather X
Rencher
Wallee X-1
X-7
R1
R1
Road Cont. -D Proiiies
10216.56
61.210.441
18.1:118
10212.06
100:0.974
90:9:10.33
100 15
7255:5 14
100:11 18
652:2267
61.1:0.51
Floed Cost. Position Deln
2441991
2766500855
96221.30
96621.71
296-100297
53.8&1.57
100622566
0.0:000
40.9:3.22
15.277.6
72.740.19
Findd Conz. -D Ist. Velusity
3.99:1.75
2461000912
718:087
834:1.13
01:0
40.7:1.355
$6.5-2
00:00
0::0
0::0
60.3:206
Plaad Cost. PD Position K2.
3.552254
16.700151
100
90991119
97 97.56007511
100 M0e
000
507439
61
10040.24
Flaed Cost. Raidon
00:00
00:00
001000
00:00
00:00
00:00
Direct Terque Cost. High Freq (500/)))
0:30
17.2300316
1,37:0511
104:2119
503
0:10
039:5577 7.
2:00::54
00:00
Direct Torque Cost. -Low Froq. (31,25130
$4.527.15
100:1121
69.2:0062
98::2.55
30.6222.56
100
100 15
53.029.35
59.2:3.72 72
100g1194
50.742.16
EvoCuatrel (Full State)
368:1106
157=111
363:1.46
123:277
116:0.499
101:000487
10062.56;
262:8.04
1141:0.933
106:009366
16343.72
EveCuntral (Resilual State)
182: 8.58
182:1.02
97:0..1
178:1.14
212:4..5
99.2:00004
100:2.966- 96c-1
-71-7.75
1.6:
104-1.1
165:2.89
EveConttel (Target + Proprics)
319:16,1
168:1.41
164:510
165:1.77
165:4944
9177001177
100122 Moe
255-7411
W.::351
105:00.776
145:1186
EvoContrel (Target)
2931133
162:1155
272:1.84
164118
205-5.04
99.62000777
10082.56
255-6333
112::0.855
105-0.7
1511244
EvoConntrl (Learned Galno)
266:11411
113:16
198-9603
150:22.5
117:10205
94.50009977
1001
299-761
116::6747
105-121
1551365
EvoCuntrel (Delia Position)
362:1288
133:1182
216:2288
119:2.78
105:0.265
101:000234
100=2.862 15
193:8841
63.5=371
90.1:2:44
14721190
Table 4: Normalized evaluation returns (R) for benchmarks trained over an equivalent number of low-level (3) environment steps (fixed environment duration of 2s per rollout). EvoControl here uses evolutionary strategy with population size (es (es_pop___t_ popppopppeee size = 64). detailed further in Appendix 1. Results, averaged over 6400 random seeds (95% confidence intervals shown), demonstratc EvoControl's 's superior normalized returns compared to baseline methods (fixed controllers and direct torque control). Scores are normalized a 0- 0-100 scale, where indicates a random policy and 100 represents the highest score achieved by any non- EvoControl baseline. Bold values highlight scores surpassing this baseline (scores 100)
Same PPO high-lesel als, p with
Ast
Hallthretah X
Hupper 12
Humanaid X
Humonoid Stamdup
Inverted Dauble Pund.
Inertal Pend.
Pusher X-:
Reacher K-
Bracher ID R.1
Walker2h
Leve Polic of
X.
R
R.1
X
X-
Food Cost. 7O Posiarn
5.0:5.11
9022:0489
92.1229
87.5:13
91:6000577
100611336-06
100:94
100 24
88.712.6
63.7:02255
Rind Cost.
Dato
4.93-167
Fioed Cost. PD lr. Velooity
594518
2.90000998
569922
724
0.:0
61.3::1.99
99:0045
0:01
0:00
9.56677772
100600681
Food Cost. Randon
00:00.0
0010000
00100
0.000.0
00:0000
0.0:0.00
0.0:0000
0.0:000
0.0:000
0.0000
0.00000
Nage
Fry (5001)))
0-00
64:01.44
S1-2.15
97 587
9:18-7.60
11.144424
52-0-122
Direct Bogge Cost. Low Fieg (51,25000
31446.52
10061.22
10061126
$7.2-1335
100:002004
100850.0178
13.1:10.2
67 07.2:3505
1001:1.7
69 092:1.31
EveCostrod (Flll State)
188::12
165:1.15
13N:32.4
111:115
90.266.
100:013306
1001:0
296:442.6
109:11.6
104:2,14
127:126
EreCostedd (Reidual State)
152:117
166:166
125:118
10R:543
112:144.
99.2:0.304
100::0
102::105
104:2.07
169:59.6
ReeCostrd (Targst + Proprin.)
183 60.2
135
130155.2
126 12.1
134 58.8
87.11:533
100:0
2.77
1:21
1.0 12.7
EveControl (Targeo)
201:27.5
164:1006
131::935
125:4453
10M:122
19:5:17
10M1:0
266:27
K3.5:634
165::13
122:-W
EveCostiol (Leumeed Gaino
103::331
901:0:6
141::845
111:259
102:1.5
93:9=25
100:00
263::66.5
1042:114
1022:485
149:35.2
EveCostral Delta Position)
168:200 29.9
123:75.2
95.011408
104:9.23
94.215.88
100::04491
100:0
308:66.6
591-3.33
99.917.077
120:
tioned obstacles penalizing collisions, explicitly designed to test the value of rapid, high-frequency control. Environment implementation details are provided in Appendix E.
for three training seeds per baseline. Results are reported as the mean normalized score R (Yu al., 2020) across all 384 evaluation rollouts (3 training seeds 128 evaluation rollouts), scaled from 0 (random policy performance) to 100 (best non-ExoControl baseline)-detaine in Appendix H.
Benchmark Methods. We compare EvoControl against established baselines, using the same high-level PPO policy (p) learning algorithm across all, varying only the low-level policy (3). We consider fixed controllers: PD Position, PD Position Delta, PD Integrated Velocity and PD Position & K2 (Aljalbout et al., 2024); direct torque control at both high (500H2) and low (31.25Hz) frequencies (Chen et al., 2023); a Random policy (30H2); and several EvoControl ablations with varying state information provided to the low-level neural network controller (Table 2). Here, the EvoControl variants using position-basee controllers are annealed from their corresponding PD controllers. Method implementation details are provided in Appendix F.
5.1. Main Results
We evaluated all benchmark methods across our environ- ments, with results tabulated in Table 3 and Table 4, for the same number of high-level policy steps (p) and the same number of low-level environment steps, respectively. Evo- Control, on average, achieves higher normalized evaluation return R on all environments. Specifically, EvoControl can both achieve high average return R while learning a slow (31 11.25Hz) high-level policy and be able to solve an envi- ronment task with a fast (500Hz) learned low-level policy. Furthermore, EvoControl can outperform direct torque con- trol at high frequency, and outperform the same high-level policy learning algorithm with position PD controllers, and we provide insights in Section 5.2. Moreover, we also show that EvoControl can support other ES for optimizing the low-level policy in Appendix J.L.
Evaluation. Unless otherwise stated we train each pol- icy (high-level and low-level (8) for IM high-level steps. Post-training, we evaluate performance using 128 rollouts (different random seeds) per trained policy, calculating the return for each 1,000-step episode. We repeat this process
7

---

EvoControl: Multi- Frequency Bi-Level Control for High-FFeqqenny Continuous Control
8 85020
Pope dat.
Table 5: Normalized evaluation returns (R) on the Safety Critical Reacher environment, using the same normalization from Table 3. EvoControl effectively learns a low-level controller capable of rapid, adaptive responses at a high frequency, surpassing the fixed- frequency limitations of traditional PD position controllers and enabling quicker reactions to unexpected collisions. In contrast, baseline controllers at lower frequencies fail to detect collisions quickly enough, resulting in substantially lower performance.
1030
3.5
-2.4
02
0.0 (radand)
0.2
04
0.6
-20
-40
Same PPO high-level alg, with a Low-Level Policy | of
Safety Critical Reacher
10
R f
-10
70 Fotlias
Fixed Cont. PD Position
100
0.0
2.2
0.4
0.8
0.8
1.2
Fixed Cont. - PD Position Delta
0++
(.Plllyy Stepet
Fixed Cont. -PD Int. Velocity
67.1+11.4
Figure 2: Top 3 Sub-Plots: State visitation histogram for
Fixed Cont. Randdm
0.0++0.0
Reacher 1D. Empirically demonstrating that PPO with high- frequency-dire direct torque control suffers from less efficient ex- ploration compared to using PPO at freeuueuency with a hhed-higg-frequene PD controller. EvoControl can achieve the same efficient exploration as a PD controller. Bottom: Evaluation return R versus P policy steps on Reacher ID. PPO at high-frequency with direct-torque control expe- riences slower convergence compared to learning PPO at a low-frequency with fixed PD controller. EvoControl can evolve its lower-level controller throughout training. leading to a higher evaluation reward in comparison additional plots provided in Appendix I.I.
Direct Torque Cont. High Freq. (500Hz)
0::0
Direct Torque Cont. - Low Freq. (31.25Hz)
35.9=17.1
EveControl (Full State) EvoControl (Residual State) EveControl (Target + Proprio.)
205114.1
12.
237
EvoControl (Target)
121: 24.9
EveControl (Learned Gains)
169 169:17.15
EvoControl (Delta Position)
113-14.66
underpinning for EvoControl's 's improved performance. Cru- cially, we observe performing direct high-frequency torque control suffers from poor state exploration, given the same number of training steps (Appendix I.1).
Can EvoControl Learn High-Frequency Interaction Control? (P2). High-frequency control can be crucial for safety-critical tasks like collision avoidance, where rapid responses to unexpected contacts are paramount. To inves- tigate such a setting, we adapted the Reacher ID environ- ment introduce random object in 25% of the episodes which block the arm from reaching its intended goal and add both an observation for any measured contact force and a reward penalty for this contact force (Safety Critical Reacher). We tabulate the normalized performance of all baselines in Table 5. We observe that EvoControl is able to observe and react faster at high-frequency to un-oodelled collisions, compared to a low-frequency policy with a fixed- high-frequency state tracking PD controller. Critically such collision detection and avoidance environment exemplifies our intuition from Proposition 2.1, that higher-frequencyy actions can achieve a higher reward. Intuitively, a well- performing policy requires a change in behavior as soon as any un-modeled collision is detected, intuitively similar to fast automatic reflexes for low-level system controller with a high-level system, beyond simple goal-state tracking. We provide experimental details in Appendix I.2.
5.2. Insight Experiments
Next, we analyze why EvoControl performs better than learning a policy with standard fixed controllers or direct torque control. We highlight the importance of controlling an environment at high frequency when required without sacrificing learning convergence ability.
Does EvoControl Possess Efficient Exploration? (P1). To explore if the benchmarked methods during training possess efficient exploration we analyze the learning curves for the Reacher ID environment and their state space visitation fre- quency histograms in Figure 2 (with implementation details in Appendix I.I.. We observe that EvoControl can initially achieve the same efficient state exploration due to temporal abstractions, similar to that of a fixed PD position controller, and then can further achieve a higher evaluation reward throughout training. This suggests that the higher-frequency control enabled by EvoControl, as theoretically motivated by Proposition 1. contributes an increased reward. This is reminiscent of the principle behind Pulse Width Modula- tion (PWM), where higher frequency allows for finer con- trol and more accurate signal representation (Huang et al., 2011). While a direct equivalence to PWM is not claimed, actions to improve control,
This finding is further supported by experiments on the higher-dimensional Safety Critical HalfCheetah task, de- tailed in Appendix J.14, which features a more complex di-dinensilnal state space. In this task, EvoControl sim-
the ability of as demonstrated in Proposition 1. provides a theoretical
8

---

EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
ilarly excels, leveraging high-frequency control to rapidly detect and adapt collisions, reinforcing the robustness and generalizability of the approach.
Table 6: Normalized evaluation returns (R) for benchmark meth- ods on the Reacher ID environment, using the same normalization from Table 3. EvoControl is more robust to the tuning of the un- derlying PD controller than existing fixed PD controllers, which can degrade as their PD controller parameter ( (Kp) becomes less tuned for the task.
Furthermore, we validated EvoControl's practicality for real- world deployment by demonstrating zero-shot sim-to-real transfer on a 7-DoF Franka Emika Panda robot for tabletop manipulation tasks. Our real-world validation showed Evo- Control can operate effectively at high-frequency (200Hz low-level control) with rapid inference (average 64 ju per step), comfortably surpassing typical robotic control rates. Importandy, EvoControl reduced collision forces compared to a tuned PD controller, highlighting benefits for safety- critical upplications. We include representative results and visualizations in Figure and Figure 4, and refer to Ap- pendix K for comprehensive details of these real-robot ex- periments.
Same PPO high-leve ale, with Low
Kp - -0.001
Kg 0..1
Kp 1.0
K2 - 10.0
R1
R1
R1
Cont Position
8:10
13.5+7.00 7.39
8188-2.81
000-0.801
Aitend Cont. Position Delta
0:10
01:0
146:27.1
08.81355
Cont Velucity
8:10
01:0
0-0
Fizeed Cont. Ranaon
0.::000
0.060.0
00:0000
0.0+0:00
Torque Cont. High Freq. (500H))
43.6:6.48
43.66545
43.626.48
43.616645
Direct Torgar Cani. LLw Frog (31 25Hz
96241.57
96.2 188
96 96.2=1.85
96.2 1.87
EvuCsntro State)
EvoCantred (Resihal State)
100--4.5
99.515.62
100-3.29
2018
EvoConted (Target Propeins)
98.6:35
97.9:1.48
101:5443
101:0.504
EvoCantred (Tangre)
94.6-6685
96.5+10.1
101-1.58
100-0.995
ExoContee (Learued Cains)
91,8=5.37
93446669
101:2.35
100:3304
EvoContee (ella Position)
95.818.29
97,11992
95.4:1222
97.6:88.2
cally, evaluating on the Reacher 1D task, with varying the K₁ = {0.001, 0.1, 1.0, 10.0} gain, we observe EvoControl achieving a higher average normalized return R than a PD controller, crucially as the PD controllers become less tuned Kp + 0 their performance decreases, highlighting the sensi- tivity of these fixed controllers. This highlights EvoControls robustness to PD parameters, which could arise due to the PD controller providing a semantically meaningful initial latent action 0g that the lower-level policy can then refine. Ablating the initial PD controller annealing in EvoControl destabilizes learning of both the high-level and low-level policies, confirming its importance (Appendix J.5).
Figure 3: Real-World Robot Setup. Left: MuloCo simulation of the Franka Emika Panda robot setup. Right: Corresponding real hardware configuration for sim-to-real validation.
6. Conclusion and Future Work
In this paper, we present EvoControl, novel bi-level pol- icy learning framework for learning both a slow high-level policy and a fast low-level controller using PPO and ES, respectively, for continuous-control tasks. Theoretically, we show that there exist some CTMDPs in which acting at higher frequencies can yield strictly higher expected cumu- lative reward. Empirically, EvoControl outperforms existing control methods, particularly in tasks requir- ing fast reactions. Moreover the limitations of the current approach, are that EvoControl still relies on the existence of a fixed-PD controller for the task (common in robotics applications, Appendix J.5) and can use more computational complexity (Appendix J.4) compared only performing PPO, which can be readily parallelized in practice with mod- ern accelerated compute platforms, both could be readily improved. In addition, promising future directions include exploring more complex nested hierarchies, direct low-level to high-level information flow, and ensembles of policies (Appendix I.).
Figure 4: Real-Robot Tasks. Left: Block collision experiment measuring collision forces. Right: Bin-opening task demonstrating gentle high-frequency torque control.
Can EvoControl Automate Tuning of PD parameters? (P3). A widespread limitation of any Dxcd-Pe--low-evel controller is the inherent sensitivity of its state state-tracking per- formance to that of its fixed gains (Kp, K2). In practical scenarios such gains require careful manual tuning to each task and environment of operation. Therefore having an approach that can be more robust to tuning PD gains than just their inherent sensitivity is practically useful. Empiri-

---

EvoControl: Multi-Frequeeet Bi-Level Control for High-Frequency Continuous Control
Acknowledgements
International Journal of Dynamics and Control, 9:818- 827, 2021.
thank the anonymous reviewers, and area chairs, and specifically Francesco Nori, Leonard Hasenclever, Steven Bohez, Thomas Lampe, Baruch Tabanpour, Antoine Moulin, Nimrod Gileadi, Jose Enrique Chen and Taylor Howell for their insightful comments and suggestions that ultimately improved this work.
Bosman, P. A., Grahl, J., and Thierens, D. Benchmarking parameter-free amalgam functions with and without noise. Evolutionary computation, 21(3):445-44) 469, 2013.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas,
Impact Statement
J., Wanderman-Milmn S., et al. Jax: Autograd and xla. Astrophysics Source Code Library, pp. ascl-2111, 2021.
Our novel bi-level policy learning framework, which trains fast-reacting low-level controller alongside slower high- level policy, offers the promise of safer, better control in areas such as surgical robotics, autonomous driving, and industrial automation. By enabling high-frequency decision- making, it can better handle unexpected events and reduce risks. However, leveraging learned control for safety-critical systems requires rigorous testing and transparency to pre- vent unexpected failures behaviors. Organizations adopt- ing this technology should address potential biases, priori- tize robustness, and maintain thorough oversight ensure reliable, ethical deployment.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016a.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J.,
Schulman, J., Tang. J., and Zaremba, W. OpenAl gym. CoRR, (bb//606.01140, 2016b.
Brunton, S. L., Proctor, J. L., and Kutz, J. N. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932-3333 2016.
References
Chen, B., Xu, M., Li, L., and Zhao, D. Delay- aware model-
based reinforcement learning for continuous control. Neu- rocomputing. 450: 12- 2021a.
Abramowitz, S. and Nitschke, G. Scalable evolutionary hierarchical reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 272-275, 2022.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 144154-555097, 2021b.
Aljalbout, E., Frank; F., Karl, M., and van der Smagt, P. On the role of the action space in robot manipulation learning and sim-to-reat transfer. IEEE Robotics and Automation Letters, 2024.
Chen, S., Zhang, B., Mueller, M. W., Rai, A., and Sreenath, K. Learning torque control for quadrupedal locomotion. 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pp. 1-8. IEEE, 2023.
Bäck, T., Foussette, C., and Krause, P. Contemporary evolu- tion strategies, volume 86. Springer, 2013.
Bacon, P.-L., Harb, J., and Precup, D. The option-crite architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Chentanez, N., Müller, M., Macklin, M., Makoviychuk, V., and Jeschke, Physics-based motion capture imitation with deep reinforcement learning. In Proceedings of the 11th Annual International Conference on Motion, Interaction, and Games, pp. 1-10, 2018.
Battaglia, P. W., Hamrick, B., Bapst, V., Sanchez- Gonzalez, A., Zambaldi, V. Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al. Rela- tional inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 01261, 2018.
Chi, C.,, Xu, Z., Feng. S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. Diffusion policy: Visuomo- tor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273668. 2023.
Bodnar, C., Day, B., and Lió, P. Proximal distilled evolu- tionary reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2283-3290,22 2020.
Chiaverini, S., Siciliano, B., and Villani, L. A survey of robot interaction control schemes with experimental com- parison. JEEE/ASME Transactions on mechatronics, (3):273-285, 1999.
Borase, R. P., Maghade, D., Sondkar, S., and Pawar, S. A review of pid control, tuning methods and applications.
10

---

EvoControl: Multi- Fuequ-Fre Bi-Level Control for High-Frequency Continuous Control
Conti, E., Madhavan, V., Petroski Such, F,, Lehman, J., Stanley, K., and Clune, Improving exploration in evo- lution strategies for deep reinforcement learning via population of novelty-seeking agents. Advances in neural information processing systems, 31, 2018.
Hogan, N. Impedance control: An approach manipulation.
In 1984 American control conference, pp- 304-313. IEEE, 1984.
Holt, S., Hüyükk, A., Qian, Z.., Sun, H., and van der Schaar, M. Neural laplace control for continuous-time delayed systems. In International Conference on Artificial Intelli- gence and Statistics, pp. 1747-1778. PMLR, 2023a.
Dabney, W., Ostrovski, G., and Barreto, A. Temporally- extended e-greedy exploration, 2020.
Dantec, E., Taix, M., and Mansard. N. First order approx- imation of model predictive control solutions for high frequency feedback. IEEE Robotics and Automation Let- ters, 7(2):4448 4455, 2022.
Holt, S., Hüyük, A., and van der Schaar, M. Active ob- serving in continuous-time control. Advances in Neu- ral Information Processing Systems, 36:46054-46092. 2023b.
Davchev, T., Luck, K. S., Burke, M., Meier, F., Schaal, S., and Ramamoorthy, S. Residual learning from demonstra- tion: Adapting dmps for contact-rich manipulation. IEEE Robotics and Automation Letters, 7(2):4488 44945, 2022.
Holt, S., Luyten, M. R., Berthon, A., and van der Schaar, M. G-sim: Generative simulations with large language models and gradient-free calibration. In Forty-second International Conference on Machine Learning. 2025.
Dayan, and Hinton, G. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
Huang, J., Padmanabhan, K., and Collins, O. The sam- pling theorem with constant amplitude variable width pulses. IEEE Transactions on Circuits and Systems I: Regular Papers, 58(6): 11t--1178911,, 2011.
Elfwing, S., Uchibe, E., Doya, K., and Christensen, H. Evolutionary development of hierarchical learning struc- tures. IEEE transactions on evolutionary compulation, 11(2):249-244, 2007.
Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and AraÅšjo, J. G. Cleanrl: High-quality single- fille--iieeee implementations deep reinforcement learn- ing algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Findeisen, W. Bailey, F. N., Brdys, M., Malinowski, K., Tatjewski, P., and Wozniak, A. Control and coordination in hierarchical systems. John Wiley & Sons, 1980.
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., and Levine, S. Residual reinforcement learning for robot control. In 2019 international conference robolics and automa- tion (ICRA), pp. 6023-6029. IEEE, 2019.
Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2/06. 1, 2021.
Kacprzyk, K., Holt, S., Berrevoets, J., Qian, Z., and van der Schaar, M. Ode discovery for longitudinal het- erogeneous treatment effects inference. arXiv preprint arXiv:2403. 2024.
Garcia, E. and Antsaklis, P. J. Model-basee went-triggered control for systems with quantization and time-varying network delays. IEEE Transactions on Automatic Control, 58(2):422 434, 2012.
Khadka, S. and Tumer, K. Evolution-guidee policy gradient in reinforcement learning. Advances Neural Informa- tion Processing Systems, 31, 2018.
Guo, J., Kurup, U., and Shah, M. Is safe to drive? an overview of factors, metrics, and datasets for driveability assessment in autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 2133)33135-3151, 2019.
Kingma, D. and Ba, Adam: method for stochastic optimization. International Conference on Learning Rep- resentations, 12 2014.
Hansen, N. and Ostermeier, A. Completely derandomized self-adaptation in evolution strategies. Evolutionary com- putation, 152)159-195, 2001.
Kuchenbecker, J., Gewirtz, J., McMahan, W., Standish, D., Martin, P., Bohren, J., Mendoza, P. J., and Lee, D. I. Verrotouch: High-frequency acceleration feedback for telerobotic surgery. In Haptics: Generating and Per- ceiving Tangible Sensations: International Conference, EuroHaptics 2010, Amsterdam, July 8-10, 2010. Proceed- ings, Part I, pp. 189-196. Springer, 2010.
Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., and Silver, D. Learning and transfer of modulated locomotor controllers. arXiv preprint 160.0.66..0 2016.

---

EvoControl: Multi- Freti-Frequency Bi-Level Control for High-Frequency Continuous Control
Lange, R. T. evosax: Jax-based evolution strategies. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pp. 639-662, 2023.
Congress Evolutionary Computation (CEC), pp. 1-8. IEEE, 2022.
Norris, J. R. Markov chains. Number 2. Cambridge univer- sity press, 1998.
Li, P., Hao, J., Tang, H., Fu, X., Zhen, Y., and Tang, K. Bridging evolutionary algorithms and reinforcement learning: A comprehensive survey on hybrid algorithms. IEEE Transactions Evolutionary Computation, 2024.
Oh, S., Woo, H., and Kong, K. Frequency-shaped impedance control for safe human-robot interaction in reference tracking application. JEEE/ASME Transactions On Mechatronics, 19()):1977-116,, 2014.
Li, S., Zhang.J., J., Wang. J., Yu, Y., and Zhang, C. Active hi- erarchical exploration with stable subgoal representation learning. arXiv preprint arXiv:2 105. 14750, 2021.
Oku, E., Obot, E. P., and Author, Comparative study of pd, pi and pid controllers for control of single joint system in robots. Int. J. Eng. Sci, V2)) 2018.
Lu, C., Kuba, J., Letcher, A., Metz, L., Schroeder de Witt, C., and Foerster, J. Discovered policy optimisation. Ad- vances in Neural Information Processing Systems, 35: 16455-16468, 2022.
Parr, R. and Russell, S. Reinforcement learning with hi- erarchies of machines. Advances neural information processing systems, 10, 1997.
Luo, J., Solowjow, E., Wen, C., Ojea, J. A., and Agogino, A. M. Deep reinforcement learning for robotic assembly of mixed deformable and rigid objects. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pp. 2062-200220000 2069. IEEE, 2018.
Pateria, S., Subagdja, B., Tan, A.-h., and Quek, C. Hierar- chical reinforcement learning: A comprehensive survey. ACM Computing Surveys (CSUR), 54(5):1-35, 2021.
Peng. X. B. and Van De Panne, M. Learning locomotion skills using deeprl: Does the choice of action space mat- ter? In Proceedings of the ACM SIGGRAPH//urogrup ics Symposium on Computer Animation, pp. 1-13, 2017.
Ma, Y. Song, Z., Zhuang. Y., Hao, J., and King, I. A survey on vision-language models for embodied ai. arXiv preprint arXivv2405. 2024.
Martin-Martin, R., Lee, M. A., Gardner, R., Savarese, S., Bohg, J. and Garg, A. Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks. In 2019 IEEE/RSJ inter- national conference on intelligent robots and systems (IROS), pp. 1010-1017. IEEE, 2019.
Peng, X. B., Abbeel, P., Levine, S., and van de Panne, M. DeepMimic: Example- -uidedg deep reinforcement learn- ing of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4):143 2018.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselinest. Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http: html.
McClinton, W., Levy, A., and Konidaris, G. Hac explore: Accelerating exploration with hierarchical reinforcement learning, arXiv preprint
//imlr. www..org/ppaee wwtppppppes//2//200-00oe/
2021.
Rao, D., Sadeghi, F., Hasenclever, L., Wulfmeier, M., Zam- belli, M., Vezzani, G., Tirumala, D., Aytar, Y., Merel, J., Heess, N. M. O., and Hadsell, R. Learning transfer- able motor skills with hierarchical latent mixture policies. ArXiv, abs/2 12.05062. 2021.
Mundhenk, T. N., Landajuela, M., Glatt, R., Santiago, C. P., Faissol, M., and Petersen, B. K. Symbolic regres- sion via neural-guided genetic programming population seeding. arXiv preprint arXiv:21 2021.
Nachum, O., Gu, S., Lee, H., and Levine, S. Data-efficient hierarchical reinforcement learning. In NeurIPS, 2018a. Sa.
Rechenberg, I. Evolutionsstrategie.. Optimierung technis- cher Systeme nach Prinzipien derbiologischen Evolution, 1973.
Nachum, O., Gu, S., Lee, H., and Levine, S. Data-efficient hierarchical reinforcement learning. arXiv preprinl arXiv:1805.08296, 2018b.
Reda, D., Tao, T., and van de Panne, M. Learning to loco- mote: Understanding how environment design matters for deep reinforcement learning. Proceedings of the 13th ACM SIGGRAPH conference on motion, interaction and games. pp. 1-10, 2020.
Nachum, O., Tang, H., Lu, X., Gu, S., Lee, H., and Levine, S. Why does hierarchy (sometimes) work so well in re- inforcement learning? arXiv preprint arXiv:1909.10611 2019.
Reiter, R., Hoffmann, J., Reinhardt, D., Messerer, F,, BaumgAIrtner, K., Sawant, S., Boedecker, J., Diehl, M., and Gros, S. Synthesis of model predictive control and
Nomura, M. and Ono, Fast moving natural evolution strategy for high-dimensional problems. In 2022 IEEE
12

---

EyoControl: Multi-Frequeeie Bi-Level Control for High-Frequency Continuous Control
reinforcement learning: Survey and classification, arXiv preprint ar arXiv:2502.02/333 2025.
Sutton, R. S., Modayil, J.. Delp, M., Degris, T., Pilarski, P. M., White, A., and Precup, D. Horde: A scalable real- time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Sys- tems - Volume 2, AAMAS '11, pp. 761-768. International Foundation for Autonomous Agents and Multiagent Sys-
Riedmiller, M., Hafner, R., Lampe, T., Neunert, M., De- grave, J., van de Wiele, T., Mnih, V., Heess, N., and Springenberg, J. T. Learning by playing solving sparse reward tasks from scratch. In Proceedings of the 35th International Conference on Machine Learning, 2018.
tems, 2011, ISBN 0982657161.
Ros, R. and Hansen, N. A simple modification ema-es achieving linear time and space complexity. In Inter- national conference on parallel problem solving from nature, pp. 296-28666 3006030556600 Springer. 2008
Tang. Y. Tian, Y., and Ha, D. Evojax: Hardware-accellen neuroevolution. In Proceedings of the Genetic and Evo- lutionary Computation Conference Companion, pp. 308- 311, 2022.
Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, I. Evolution strategies as a scalable alternative to rein- forcement learning. arXiv preprint 1733:17033.3864 2017.
Team, O. E. L., Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., et al. Open-ended learning leads to gener- ally capable agents. arXiv preprinl arXiv:2107. 12808 2021.
Salter, S., Wulfmeier, M., Tirumala, D., Heess, N., Ried- miller, M., Hadsell, R., and Rao, D. Mo2: Model-based offline options. In Conference on Lifelong Learning Agents. pp. 902-919. PMLR, 2022.
Vasic, M. and Billard, Safety issues in human-robot interactions. In 2013 ieee international conference on robotics and automation, pp. 197-204. IEEE, 2013.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 17X7.v/707.0665 2017.
Venkataraman, S. and Gulati, S. Terminal slider control of robot systems. Journal of Intelligent and Robotic Systems, 7:31-55, 1993.
Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Pe- ters, J., and Schmidhuber, J. Parameter-exploring policy gradients. Neural Networks, 23(4):551-555 2010.
Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jader- berg, M., Silver, D., and Kavukcuoglu, K. Feudal net- works for hierarchical reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 3540-3549. PMLR, 2017.
Sigaud, O. Combining evolution and deep reinforcement learning for policy search: A survey, ACM Transactions on Evolutionary Learning, 3(3):1-20, 2023.
Vezzani, G., Tirumala, D., Wulfmeier, M., Rao, D., Abdol- maleki, A., Moran, B., Haarnoja. T., Humplik, J., Hafner, R., Neunert, M., al. Skills: Adaptive skill sequenc- ing for efficient cemporally-extentte exploration. arXiv preprint arXiv:2211. 13743, 2022.
Song, P.. Yu, Y., and Zhang, X. A tutorial survey and com- parison of impedance control on robotic manipulation. Robotica, 37(5):801-83) 2019.
Storn, R. and Price, K. Differential evolution-a simple and efficient heuristic for global optimization over continu- ous spaces. Journal of global optimization, 11:341-359, 1997.
Wahlström, N., Schön, T. B., and Deisenroth, M. P. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv:1502.0225: 2015.
Watter, M., Springenberg, J., Boedecker, J., and Riedmiller, M. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
Suri, K., Shi, X. Q.. Plataniotis, K. N., and Lawryshyn, A. Maximum mutation reinforcement learning for scalable control. arXiv preprint arXiv:2007.13690 2020.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Wei, B. A tutorial on robust control, adaptive control and robust adaptive control applicatonni robotic manipula- tors. Inventions, 4(3):49, 2019.
Sutton, R. S., Precup, D., and Singh, S. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2); 181-211 1999.
Wierstra, D., Schaul, T., Glasmachers, T.. Sun, Y., Peters, J., and Schmidhuber, J. Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949-98) 2014.
13

---

EvoControl: Multi-Freqeecc Bi-Level Control for High-Prequency Continuous Control
Wöhlke, J., Schmitt, F., and Hoof, H. Hierarchies of planning and reinforcement learning for robot navigation. In 2021 IEEE international conference robotics and automation (ICRA), pp. 10682-10688. IEEE, 2021.
Wulfmeier, M., Abdolmaleki, A., Hafner, R., Tobias Sprin- genberg, J., Neunert, M., Siegel, N., Hertweck, T., Lampe, T., Heess, N., and Riedmiller, M. Composi- tional transfer in hierarchical reinforcement learning. Robotics: Science and Systems XVI, Jul 2020a. doi: 10.116007fs..2020.v..6555 URL http: //tp://dx.../ /xx. doi. org/10. 156.7 ras 2020. xvi.054.
Wulfmeier, M., Rao, D., Hafner, R., Lampe, T., Abdol- maleki, A., Hertweck, T., Neunert, M., Tirumala, D., Siegel, N., Heess, N., and Riedmiller, M. Dffa-efficient hindsight off-policy option learning, 2020b.
Xie, 7.., Clary, P., Dao, J., Morais, P., Hurst, J., and Panne, M. Learning locomotion skills for cassie: Iterative design and sim-to-real. Conference on Robot Learning. pp- 317-329. PMLR, 2020.
Yang. X., Ji. Z., Wu, J., Lai, Y.K.., Wei, C., Liu, G., and Setchi, R. Hierarchical reinforcement learning with uni- versal policies for multistep robotic manipulation. IEEE Transactions on Neural Networks and Learning Systems, 33(9):7277-744 2021.
Yu, T., Thomas, G., Yu, L., Ermon. S., Zou, J., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.
Zakka, K., Tabanpour, B., Liao, Q. Haiderbhai, M., Holt, S., Luo, J. Y., Allshire, A., Frey, E., Sreenath, K., Kahrs, I.. A., et al. Mujoco playground. arXiv preprint arXiv:2502.08844 2025.
Zheng. H., Wei, P., Jiang. J., Long, G., Lu, Q., and Zhang, C. Cooperative heterogeneous deep reinforcement learning. Advances in Neural Information Processing Systems, 33: 17455-17465, 2020.
14

---

EvoControl: Multi- Freqi-Frequenc Bi-Level Control for High-Prequency Continuous Control
Appendix
Table of Contents
A Expanded Problem
17
B Proof Proposition Existence CTMDPs Requiring Higher-Frequency Control Optimality
B.I Intuitive Continuous Control Safoty-Critical Critical Example
20
C Expanded Background: Fixed PD Controllers
20
Extended Related Work
21
E Environment Selection and Implementation Details
23
Standard Gym MuluCo Tasks
E.2 Reacher ID
24
Safety Critical Reacher
E,4 Safety Critical HalfCheetah
25
Low-Level Controller (PD) Parameters
F
Benchmark Method Implementation Details
28
El
High-Leveel Policy and PPO Implementation
28
F2 PD Controller Implementation.
29
F3 Fixed Controllers
29
F44 Direct Torque Control
29
G EvoControl Implementation Details
30
G.1 EvoControl PseudoCode
.eardoCCoo......................
30
G.2 Detailed Analysis of EvoControl
31
G.3 Computational Considerations.
32
H Evaluation Metrics
33
Additional Experimental Setup
33
I.1 Efficient Exploration Experimental Sctup
33
1.2 High-frequency Interaction Control in Safety Critical Reacher
.e..............................
33
J Additional Experiments
33
J.1 EvoControl Supports Other Competitive ES "oo.............................
33
J.2 Ablation: Using PPO to Train the Lower-level Policy
................................
35
J.3 EvoControl Outperforms Direct Torque Control High ................................
35
J.4 Ablation: Equal Computational Complexity for All Baselines
36
J.5 Ablation: No annealing with PD Controller
41
J.6 Ablation: Main Results for More High-level Steps
42
J.7 Main Table of Results Additional Metries
43
J.8 Learning Curves for All Baselines
J.9 Ablation: Training High-Level Policy with Evolution Strategies
48
J.10 Rollout Trajectory Plots of High-level Action Baselines
49
15

---

EvoControl: Fretu-eryy Bi-Level Control for High-Prequency Continuous Control
J.11 Ablation: Removing Communication Between the Layers in EvoControl.
53
J.12 Ablation on Varying Fixed Values of the Annealing Parameter
56
J.13 PD Position Delta Action Scaling Sweep
J.14 Safety Critical HallCheetah
57
J.15 Comprchensive Baseline Comparison
58
K Real-World Validation on Franka Emika Panda
59
L Limitations & Future Work
61
Reproducibility Statement
61
N Common Questions and Discussion
61
Why Not Simply Pre-train with a PD Controller Imitate?
61
N.2 Relationship to Pulse-Width Modulation
61
Discussion of Theoretical Analysis on Convergence the Bi-Level Framework
61
N.4 Comparison of EvoControl with Model Predictive Control (MPC)
62
16

---

EvoControl: Multi-Frequeeet Bi-Level Control for High-Frequency Continuous Control
A. Expanded Problem
In the following we expand the problem setup from the main paper.
States & Actions. We denote the environments state space as S R and its action space as U C. R At time E F, the system's 's state is represented by 8, E S, and its action by un E U. Considering action (e.g. actuator) limits the action space is constrained to a box in Euclidean space: U = [Umin> Mmax!
Environment Dynamics. The transition dynamics for continuous control environments can be described by an underlying da.
unknown differential equation of s; or f(,,, uz). The transition function, which describes the evolution of the state over a discrete time step 4s can be approximated using the Euler method S++A, & 8t + A fst. ut) Given an action uy and current state 8t, St+A, ~ P(S++ A, |St; 44) is implicitly defined by this approximation. More sophisticated numerical integration schemes (e.g., Runge-Kutte methods) can be employed for higher accuracy. In stochastic environments, the dynamics function f can be considered to be stochastic, leading to probability distribution over next states given the current state and action. We consider the setting where there is additional observation function that maps the current environment state to an underlying observation = g({t) + c, where is optional observation noise, e.g. Gaussian noise with zero mean E, ~ o0) " To simplify notation we use observation and state interchangeably and clarify the specifics when needed.
Policies. The agent can be represented as a single policy π TR.. + Rd,,, that observes the current observation time and samples an action i, ~> (8t) and then applies this action the environment a given fixed A.. In the case of stochastic policy, the action is sampled from distribution conditioned on the state: u, (-1ss))
To formalize a bi-level policy, we decompose т into two components: slow, high-level policy, p. and fast, low-level policy, B. Both policies interact with the environment as described in Algorithm 1. The high-level policy operates at a lower frequency and outputs a high-level (latent) action, (g ~ p(*)), at time step k. This latent action often represents a desired high-level target, such as a target position or velocity. The low-level policy 3 operates at higher frequency and receives the high-level action ag as imput. 3 then generates the low-level actions, Ug-6, at a finer mner-graieed time step corresponding direct motor torques or other low-level control signals. These low-level actions aim achieve the high-level target specified by p. We denote the high-level time steps as k and low-level time steps distinction. The fast, low-level policy 33 operates at a frequency of 1/4., where A, is the low-level time step. The slow, high-level policy guides the low-level policy over a longer horizon. Specifically, issues a latent action which is executed by B3 for G steps. This means / effectively operates at a frequency of 1/GA,, with a time step of GA,,
to maintain this
Objective. The environment produces a reward r' sampled from an unknown reward function r(5,, ut), r + R. The overall objective of the agent is to maximize the expected future discounted reward EE To Ro where 0 S y < 1 is the discount factor.
EE 7'r
Markov Decision Process (MDP), We can model the environment as a Markov Decision Process (MDP), defined by the tuple M = (S, P, r,,,,, 7). where:
S C Rd is the continuous state space.
Ul ddd is the continuous action space.
P(s:+a, |st, un) is the state transition probability distribution, implicitly defined by the dynamics function f and the discretization scheme (e.g., Fuler method).
S SSx x U + R is the reward function, providing scalar reward re r(s,, un) at each time step.
y E [0, 1) is the discount factor, determining the importance of future rewards.
"For the standard MuloCo Brax environments use, the joint velocity observation has Gaussian noise added it, following the standard implementation of the environments (Freeman al., 2021).
17

---

EvoControl: Multi-Freqeecy Bi-Level Control for High-Prequency Continuous Control
B. Proof of Proposition 2.1: Existence of CTMDPs Requiring Higher-Frequency Control for Optimality
Proposition B.1. Consider continuous-time Markov Decision Process (CTMDP) with finite state space and action space. Let this CTMDP be discretized with time step 4.. There exist CTMDPs such that for any fixed discretization step A, > 0, there exists finer discretization A < A where the optimal policy for the A Adddiscretized MDP achieves higher expected cumulative reward over a fixed time horizon T than the optimal policy for the Ay-discretized MDP.
Full Set of Assumptions:
We consider a CTMDP with state space and action space A. When discretized with a time step A.. the transition probabilities are governed by transition rate matrix Q(a), where Qiy(a) denotes the transition rate from state to state under action The transition probabilities in the discretized MDP are then given by:
[c + +(4))) [e +
P(sy|s;, a. A,)
==
where eQ(a)s, is the matrix exponential. This is standard assumption when discretizing CTMDPs, justifiable by the Taylor expansion of the matrix exponential (see, e.g., Norris (1998), Chapter 2, Theorem 2.8.2). We assume that the error terms ( converge uniformly to zero as A, + 0. The reward function r(s, a) is the same for both the continuous and discretized cases. The time horizon T > 0 is fixed, and we assume T is an integer multiple of both A, and A,
Proof. CTMDP and MDP Definition: We consider a CTMDP with the following properties:
State Space: S {ggood 9bad
}
Action Space: A = (mmintain: Grecaver
Transition Rates:
Under Amaintain' transition rate from &good to Sbad is p > No other transitions.
Under Grecovr' instantancous transition from Stud to ggood No other transitions.
Reward Function: r(Sgood, a) = Ygoodd >
0; r r(Sbad, a)) = Fhud < O.
Discount Factor: 7 1 (undiscounted)
Episode Duration:
Initial State: Sgood
Discretizing this CTMDP with time step A, yields an MDP with transition probabilities described in the assumptions above, derived from the transition rates. Specifically, for small Ac
P( 1 goods Ommintain: Ad) R ppp
P( (8 ((oo Sgood, Omuintain A,) R -A,
P( P(Sgood Sbad, Arecovers A:) =
Cumulative Reward Definition: The cumulative reward A(TT, T, A.) for a policy x over a time horizon the MDP with discretization A, is:
T/A.1 -1
R(m, T, Ax) = E
Σ
r(st, A.
==
Proof Steps:
18

---

EvoControl: Multi-Freqeeccy Bi-Level Control for High-Prequency Continuous Control
Optimal Policies (**(A,)): For any discretization Ag, the optimal policy x² (A,) is to apply drecover in 8bad and Amaintain
in Sgood-
2) Expected Cumulative Reward (Finer Discretization A = A::: Consider finer discretization A; = Ar for some integer n > 1. Under x²( (A),, the probability of transitioning to Sbod and staying there for time k&, within A, interval approximately (*) (p Ar (*(1 [ p A ) Since recovery is immediate, the expected time 8bad within a A, interval
Ei=1 kA (+) (p A )a(1 A )m p (م,)² The expected time spent Syood is then A m(e)²² Thus the expected
reward per Ar interval is:
2(A,)2 n
gooood + p(A,)2
p(A,)?
(A,
n Tbod Aggoood
(Fgood Tbod)
n
And with 3.
such intervals, the total expected cumulative reward is:
T (DeT good p(A,)2
(A)),
(rgood rrbad))
TpAr A
n
Trgood
(Tgood Thad)
n
3) Expected Cumulative Reward (Coarser Discretization A,): For fixed D,, the stationary distribution probabilities under TT (A,) can be approximated by solving the balance equations: Mggood p21) (baa(1) Mgood and Mgood (pA,) + Phad- Also Vgood + jbod = 1. This gives /bbad p', and Pgood = 1 pD,.
Pbad (0) =
Thus, the expected reward per A, step is:
Rstep PgoodT gooo + Mbaddbad
(1 pDi)r Doood good + (pAs))bbdd
good pA&T good + paaTbad
good pA1 good Tbad)
Over the entire horizon T, with I. steps, this gives:
T A. (r (Tgood pDee(rood - Tbad ))
(A),, =
Tr good Ty4 good rbad)
4) Comparison: Comparing the rewards, we have:
R(TT T, - (A),,
TpA: (Tgood (badd) (Trgood - TppAr(rgood rbad))
gTrgood
n1 TpA (rgood -bbad) Troood + TpA, (r good rbad)
Tryood
n
TpA, (rgood Tbad) Tppeerrgoo rbad)
Tp4 (Tgood Pbadd
Since rgood? > Tbads p > 0, A, > 0. and n > 1, this difference positive. Therefore, for any As, a finer discretization A St with leads to higher expected cumulative reward under its optimal policy.
19

---

EvoControl: Multi-Frequerte Bi-Level Control for High-Frequency Continuous Control
B.1. Intuitive Continuous Control Safety-Critical Example
Consider a safety-critical task involving a one-degree-of-freeeee
robot arm. The arm's state is its joint angle 0,, and the action is the motor torque T.- The goal is to reach target angle ggoul from initial angle 9n within a fixed episode duration T. An immovable obstacle may appear in random subset of episodes (e.g., 25% of the time), obstructing the direct path to the goal. The reward function encourages reaching the target angle while penalizing contact forces with the obstacle:
1/' (Tenmtee - 10pmll Tcollision Pe dt
R R-
where ''goal and Teollision are positive weighting constants, and Fj is the magnitude of the contact force between the arm and the obstacle at time t (0 if no contact). The policy receives the observation O = (fe, de, Tt, where Tt,messured the measured torque, reflecting contact forces if any.
A standard approach might a position PD controller with low-frequency high-level policy that provides the target angle Ourget. However, this approach faces limitations. If the obstacle is present, the PD controller will exert a continuous force against it, incurring significant penalties, The low-frequency policy might only detect the collision after a substantial delay, making it difficult to react effectively.
fegguereyy policy, on the other hand, can detect the collision much faster and take corrective action. Upon detecting a sudden increase in Te.measured, can immediately reduce the motor torque, minimizing the contact force F,. Furthermore, a sophisticated high-frequency policy can learn approach the target cautiously, probing for the obstacle with small torques. If contact is detected, it can adjust its trajectory to reach the goal while avoiding further collisions.
This intuitive example illustrates how high- frequeny control can be crucial for safety-criiical tasks. It enables faster reaction to unexpected events and allows for more nuanced control strategies that consider the full reward structure, including collision avoidance. This motivates the development of methods like EvoControl, capable of effectively learning such high-frequency policies. This example highlights scenarios where tigh-frequency control offers a significant advantage over traditional orefureue control coupled with fixed controllers, especially in tasks requiring rapid responses and nuanced interaction behaviors.
C. Expanded Background: Fixed PD Controllers
Low-level PD controllers are extensively used within robotics applications, specifically when combined with a learned high-level policy p. This hierarchical structure simplifies the learning problem and effectively reduces the number of decision steps for the high-level policy. This reduction achieved by allowing the high-level policy to operate timestep of GD,,, where G is the number of low-level actions executed per high-level action, effectively reducing the number of high-level actions within fixed episode duration T.
In continuous control and robotics, these hierarchical structures, composed of learned high-level policy (p) and fixed low-level controller (e.g., a PD controller (Oku et al., 2018)), are common (Peng & Van De Panne, 2017; Song et al., 2019; Chentanez et al., 2018; Peng et al., 2018; Xie et al., 2020). The high-level policy outputs a target ag which the low-level PD controller tracks using a control signal based on the error between az and the measured system state 8t.
Commonly, PD control is designed track second-order signal, such as position or velocity. The control signal, Wr, is given by:
u (a st) + Kalâx - -e).
where Kp, Ka E IR+ are constant proportional and derivative gains, and e, = dg - 8, represents the tracking error.
Specifically for roboties, proprioceptive observed states can be represented as joint positions (q:), joint velocities (ğr), and torques (7)). This leads to several common PD control designs, summarized in Table 1. These designs differ in how the high-level target dş is interpreted and used in the control law. For instance, in "PD Absolute Position," az directly specifies the desired joint position (q while in "PD Delta Position." represents a change in joint position (89") relative to the current position. The state s; in the control law can encompass a wider range of proprioceptive information beyond just joint positions (g.). We present the target 0g and tracking error in terms of position/velocity for clarity and to align with common PD controller formulations.
20

---

EvoControl: Multi- Multu-Frequeccy Bi-Level Control for High-Freqqency Continuous Control
D. Extended Related Work
Table 7: Comparison with related bi-level learning approaches in RL. Our method, EvoControl, can achieve efficient state-action space exploration, whilst learning high-frequency interaction hehavior, and avoids tuning of PD parameters.
Approodh
Ref.
Low-ledd
High-kerd Action
High-lod Action
(PI) Elficcos
(P2) High- Frequcacy
(P3) Autontate Cauindler
y Beraard
Durattinn 3,
Explenation
Interaction Contri
Torting ISI Parameters
Fuze Costrolkas
(Serg ctsl. 2015)
Diu Rpc
[ssi]] dawaseiille
CD.
Uneect Vorger Contol
(Peng de fon Parne, 2017)
Skille
1199, (Nachernar al,
Se......)))..... A.].. 4.
chail
(e.9)
-11100 feeeefill
(Curs)
(e.2)
x- 200i
So (4)
Existing approaches to continuous control in robotics primarily fall into two categories: those employing fixed low-level controllers and those utilizing direct torque control learned end-to-end. EvoControl aims addresses limitations inherent in both approaches, and we summarize the key differences in Table 7.
Fixed Low-Level Controllers: A common strategy involves combining a learned high-level policy (often operating at low frequency, c.g., 10-30Hz) with a fixed, high-frequency low-level controller (e.g., a ProportoonalDeriiativy (PD) controller operating at 500Hz or higher) (Song et al., 2019; Chentanez et al., 8; Peng et al., 2018; Xie et al., 2020). The high-level policy generates setpoints (e.g., desired positions or velocities), and the low-level controller tracks these setpoints by adjusting actuator torques. While prevalent, this approach suffers from several drawbacks. The low-level controller's parameters require careful tuning, and its fixed nature limits its ability to handle high-frequency interactions such as unexpected collisions or disturbances. Furthermore, recent work applying RL algorithms to the physical world often restricts itself to relatively low-frequency control (~20Hz) due to the reliance on analytical impedance controllers (Martín-Marde Martín et al., 2019; Luo et al., 2018; Johannink et al., 2019). Even hierarchical approaches employing analytical controllers often limit high-level policy frequencies (Davchev et al., 2022). EvoControl, in contrast, aims to achieve efficient exploration while also enabling the learning of flexible and complex high-frequency behaviors in the low-level policy.
Evolutionary Strategies: Direct evolutionary strategies (ES approaches, (Bäck et al., 2013)) have been shown to provide an alternative for solving reinforcement learning environments; however the direct application of them, as shown by others are that they can be sample inefficient, get stuck in global minima; however excel at discovering good performing nong-horizon tasks, sparse reward tasks and delayed reward tasks, as they often optimize the episodic return, rather than the intermediate temporal difference return (Salimans et al.. 2017). There exist works formulating hierarchical ES for both levels, however still under- mader-perform gradient-based RL policy methods (Abramowitz & Nitschke, 2022). EvoControl through it's novel combination of a PPO learned high-level policy, and a ES-learndd low-level policy empirically outperforms the ablation version of using ES for both the high-level and lew-level in EvoControl, as shown in Appendix J.9. Alternatively, some methods learn end-to-end policy that directly outputs joint torques at a high frequency (Peng& & Van De Panne, 2017: Wahlström et al., 2015; Watter al., 2015). This approach, while potentially offering greater adaptability, faces significant challenges. High-frequene, control suffers from the curse of dimensionality imposed by the increased number of time steps in long horizons. The resulting explosion in the number of possible action sequences significantly hinders exploration and can lead to suboptimal policies (Martín- Martin et al., 2019; Peng & Van De Panne, 2017). EvoControl mitigates these challenges by employing a hierarchical structure, enabling more efficient exploration while retaining the adaptability afforded by direct torque control at the low level. Moreover, the related work of Peng& & Van De Panne (2017) compares learning policies with four different action spaces of direct torque control, PD position control, PD velocity control and a muscle activation's 's for the task of imitating gaits for planar walking robot environments (continuous control). Their findings correlate with ours, in that they observed on average faster learning convergence and higher task reward using low-level high- figh-freuuency (fast) controller, such as PD controller compared to performing direct torque control. Additionally Peng & Van De Panne (2017) due to having no prior controller parameters for the environments that they wanted to control, Peng & Van De Panne (2017) similarly performed a related approach where they optimized the fixed low-level controller parameters throughout training high-level policy. However, all of their low-level controllers used are simple, few parameter (2-7) controllers, such as a PD controller, and such fixed simple controllers are all only capable of sub-goal simple tracking behavior. Whereas EvoControl, can represent the fast lower-level policy with neural network policy and learn this throughout training the high-level policy, learning fast adaptive behavior of the low-level policy, that goes beyond simple sub-goal tracking behavior. Furthermore, the related work of Reda et al. (2020) studied environment design for continuous control tasks, and found that varying the control frequency of performing direct torque control in standard Mujoco Gym like environments (c.g. Ant, Hopper) could yield better learning and overall policy return, however requires tuning the control frequency (or discrete action repeats of the simulation timestep A,)
Direct Torque Control:
21

---

EvoControl: Multi- Multi-Frequency Bi-Level Control for High-Freqqenny Continuous Control
for each environment and task to get the best performance- due matching the inherent control frequency of the dynamics of the environment. They also studied the use of learning with PD controllers, and determined that PD controllers can aid in converging faster to good policy, however can get stuck in lower-reward solutions (local minima), motivating the need for a method to practically perform high-frequency torque control. In summary, EvoControl can overall learn a high-frequency policy x by learning both a slow-high-level policy p combined with a fast-low-level policy B, learning adaptive low-level behavior of an equivalent high-frequency policy, avoiding the difficulties of learning a direct torque control high-free high-frequency policy directly. Furthermore, we provide empirical evidence for the difficulty of learning direct torque control nigh-ffreqeecy policy, as even with an ever increasing number of training steps. such policy may converge to local minima Appendix J.3.
Hierarchical Reinforcement Learning (HRL): EvoControl draws inspiration from the HRL paradigm, which decomposes complex tasks into simpler subtasks managed by separate policies. Existing HRL methods such as options frameworks (Sutton et al., 1999; Bacon et al., 2017) and hierarchical actor-critic architectures (Riedmiller et al., 2018; Vezzani et al., 2022) have been successfully applied to improve exploration and learning efficiency. However, these methods typically focus on discrete skill selection or subgoal decomposition (Heess et al., 2016), while EvoControl explicitly addresses the challenges of learning low-level controller for continuous high-frequency control, enabling semantically meaningful exploration in different control modes. Related work in the RHPO/HO2M02/HLLMS family (Rao et al., 2021; Salter et al., 2022; Wulfmeier et al., 2020a;b) has also explored hierarchical approaches. However, unlike typical HRI. which focuses on skill discovery, EvoControl targets learning a fast low-level policy that complements the slow high-level policy. Uniquely, EvoControl combines PPO and ES within its hierarchical framework for efficient exploration and complex high-frequency control.
Hybrid Combinations of RL and ES: Existing related work has looked into combining evolutionary strategies ES to improve RL algorithms, specifically using them to collect diverse data as ES methods show superior exploration capabilities compared to on-policy and off- policyllcy RL algorithms, and also take updates for the RL agent itself (Sigaud, 2023; Suri et al., 2020; Khadka& & Tumer, 2018; Conti et al., 2018: 8: Li et al., 2024; Bodnar et al., 2020). Specifically, Suri et al. (2020); Khadka & Tumer (2018) use RL (SAC/DDPG) with ES data collection to collect diverse trajectories into the replay buffer to update the RL. agent. Suri et al. (2020) propose automatic mutation tuning to improve the ES component, and demonstrate improved performance on 10 out of 15 continuous control environments compared to the equivalent RL method baselines. Khadka & Tumer (2018) () also uses ES (Neuroevolution) to collect diversified trajectories, and use these trajectories in a replay buffer to train an off- off-policy RL agent. They further, update the ES data collection agent with snapshots of the trained RL agent throughout training, and demonstrate on continuous control tasks that this can lead to higher reward evaluation and faster convergence in ligher-dimensiioal state-action challenging environments. Furthermore, Zheng et al. (2020) proposes a transfer approach that has a pool of agents containing three classes of agents: on- on-policy agents, off- off-policy agents, and a population-based ES agents. All agents explore and collect trajectories into a replay buffer, with the on-policy and ES agent initially transferred from the weights off-policy (global) agent; the trajectories of the on- on-policy agents are then more frequently sampled when used to update the off-policy global agent, and use a threshold to control the frequency of policy parameter transfer. There also exist alternative solutions to combine ES and RL, such as seeding ES with an RL agent for symbolic regression (Mundhenk et al., 2021). Moreover, Elfwing et al. (2007) proposed task decomposition method, using MAXQ. to break down a complex task into hierarchy of subtasks on small dimensional state-action space problems, and used a genetic programming algorithm to learn the hierarchical task decomposition automatically. Specifically in Elfwing et al. (2007), each hierarchy corresponds to a different subtask that can be performed. Unlike EvoControl, all of these related works do not consider the problem of learning at higher frequencies, operating their continuous control environments at default large discrete time steps (e.g. 20- 10010 (Brockman et al., 2a)), and do not focus on an hierarchical approach of having high-level policy outputting latent action and a low-level policy following this latent action for G steps (providing temporal action whicractions)-whchh limits the practical deployment of their agent, as if rich features are used as inputs to the agent such images or the use of larger architectures, such as Vision Language Transformers, the inference time of the agent would increase (e.g. R 30 for images from video), limiting the agents use where high-frequency control is necessary for an environment. Conversely, EvoControl enabled from its hierarchical structure decomposing a high-level policy and a low-level policy, the low-level policy can remain a simple neural network agent of a smaller size being able to run with a fast inference time, and hence fast control operation of 500 - 1K Hz, and still gain the benefit of having a higeer-level policy that can still take as input rich features arriving at lower frequeccy such as images.
Continuous-Time Control & Planning: A surge of recent work tackles reinforcement learning in continuous-time settings where observations arrive irregularly and actions experience latency. Neural Laplace Control marries a Laplace-domain
22

---

EvoControl: Multi-Frequency Bi-Level Control for High-FFeqqenny Continuous Control
dynamics model with model predictive control (MPC) to handle unknown delays (Holt al., 2023a). Complementarily. Active Observing in Continuous- Time Control shows that when to measure the state is as critical what to do, proving that irregular sampling can strictly outperform uniform sensing while remaining computationally tractable (Holt et al., 2023b). These ideas resonate with event-triggered control in multi-agent systems (Garcia & Antsaklis, 2012) and with delay-robust model-based RI. methods, e.g. Delay-Aware Model-Based RI (Chen et al., 2021a). For long horizons, sequence-modelling planners-includdig Decision Transformer (Chen et al., 2021b) and diffusion-baset policy refinement (Chi et al., -prvierrovide strong baselines that can be embedded inside MPC loops, yielding hybrid RL-MPC schemes with favourable stability guarantees (Reiter et al., 2025). While orthogonal to EvoControl, continuous- time control methods naturally connect to high-frequency control- continuous-time behavior. Future extensions of EvoControl could incorporate these continuous-time approaches, enabling extremely fast-reacting low-level neural policies that outperform traditional MPC or planning methods constrained by computational budget at very high control frequencies.
the discretization timestep brings discrete-time methods closer to
Interpretable Dynamics and Generative Simulation: Replacing black box predictors with symbolic or sparse repre- sentations improves transparency and extrapolation. SINDY uncovers compact governing equations from data (Brunton et al., 2016), and recent work extends this philosophy to causal inference, which learns closed-form differential systems that remain valid under irregular sampling, enabling counterfactual poliey evaluation (Kacprzyk et al., 2024). Beyond inference, high-fidelity simulators are indispensable for safe planning. Recent work automates simulator construction by letting a large language model propose causal structure which is then empirically calibrated via riagdiieett-fie optimisation, achieving robust generalisation beyond historical support (Holt et al., 2025). Such approaches complement physics-grounded generative agents (Battaglia et al., 2018) and underscore the trend toward marrying interpretable models with powerful planners for trustworthy, data-efficient control. In relation to EvoControl, leveraging automated simulator construction enables the generation of diverse, open-ended training environments (Team et al., 2021), facilitating scalable online policy learning. Future versions of EvoControl could integrate these capabilities, significantly enhancing the approach's 's applicability to more complex, varied tasks and objectives.
E. Environment Selection and Implementation Details
Benchmark environments. We compare against ten standard continuous-control environments (Brockman et al., 2016a; Freeman et al., 2021), and also a safety critical continuous control environment. Specifically we use the continuous control suite from Brax (Freeman et al., 2021), which consists of ten standard continuous control environments, such as locomotion based robot control tasks such as Ant, HalfCheetah and larger state-action space environments such as Humanoid (e.g. controlling humanoid robot with state-action dimension of 60 to walk forwards with a given velocity). All the Brax environments are released under the Apache-2.0 license. Furthermore, within this standard suite of tasks is manipulation based environments such as Reacher and Pusher, where pusher is a 7 degree of freedom (DOF) robotic arm, with the task to push a movable object on a table to a desired goal location. Moreover, we also construct a safety inspired environment, adapting a single arm version of the Reacher environment, where introduce a random un-modeled contact that incurs a large negative reward when the robot manipulator collides with the object. To compare the frequency element, we set the frequency of each environment to 500Hz, and then motivated by low-lever controller running lower frequency such as 31.25Hz (G- = 16) (a realistic assumption when involving cameras to determine state), we set this as the low-level frequency.
E.1. Standard Gym MuJuCo Tasks
We use Brax (Freeman al., 2021), differentiable physics engine, which provides efficient implementations of the Ant, HalfCheetah, Hopper, Humanoid, HumanoidStandup. InvertedDoubleeenddllum Pusher, Reacher, and Walker2d environments. These environments encompass a range of locomotion and manipulation tasks, providing a diverse testbed for evaluating EvoControl. For each environment, we set the simulation timestep A, to 0.002 (500Hz operation). High-level policies operate at a frequency of 31.25Hz, achieved by executing each high-level action for G = 16 simulation steps. To ensure a fair comparison across different control modes, we remove the action magnitude penalization from the default reward function of each environment. The low-level policy receives the high-level action concatenated environment observation state as its own observation, and the exact input specification for each EvoControl variation is provided in Table 2. This allows the lowelevel controller to condition its actions the target specified by the high-level policy. The low-level action space is the same as the high-level action space. All environments have a fixed episode length 'The Brax continuous control environments all publicly available from https: / hitpbb.tttugg/ggg/g////ggg..hh com/ monn/goglee 1e/ brax.
subset of the
23

---

EvoControl: Multi-Frequenee Bi-Level Control for High-Frequency Continuous Control
of low-level timesteps of 1,000 environment steps. To increase the realism of the simulation, we run the Brax environments with the backend of MJX, that is a MuJoCo environment in Jax with XI.A. This enables to even modify the MuJoCo xml definition file (to create the Safety Critical Reacher) environment. For all MuJoCo environments, we incorporated fixed PD controllers. We tuned the PD gains for each environment individually. Specifically, we set the proportional gain (K₂) to 1.0. This value was chosen as the environments, by default, accept actions with a magnitude of 1, representing a normalized torque input. To determine the optimal derivative gain (Ka), we leveraged MuJoCo's dampratio parameter, setting it to 1.0 (critically damped), We then empirically observed the Ka value that corresponds to this dampratio within the simulation. These tuned Kp and Kg values were used consistently throughout our experiments unless explicitly stated otherwise, providing standardized and well-tuned PD baseline for comparison with EvoControl. This approach ensured that the PD controllers were appropriately configured for each environment's 's dynamics, providing strong benchmark for evaluating the performance of learned low-level policies.
E.2. Reacher 1D
The Reacher ID environment is simplified version of the standard Reacher environment. We remove the second arm link, creating a IDOF task suitable for detailed analysis. The goal is randomly placed within the reachable workspace of the single arm link. The high-level state space consists of the angle and angular velocity of the arm, and the 2D position of the target. The high-level action is the desired angle. The low-level state comprises the high-level state concatenated with the high-level action, and the low-level action is the torque applied to the joint. To ensure reproducibility we provide the full environment MuJoCo xml specification below.
canjora machter Id">
ccompiler raglen" incriairomoon "t
odefaclts
<jeinl ormetere="1"" denping="1.0 finited""ure"""
sgeom consfrinity="0 contype="" crletionn'l 0.3 0.17 12.4-"0.4 0.33 0.26 1.07s
<idefauli>
coptica gravity="O timestep="0.000 1>
apos
<nomeric date="1000 1000" omee" constraint stiffoss "tiffness" <nomerit datz="1000" oomea"costtaile limit stiffness' coomeric datz=') 6.1" mmre cosstralnt vol damping"> "> <oumztic dats="O.1" conttrsist aa daupiig Vs <nomztic date="9.."" ome--ang dan_einppe""""">>>> ">> coumeriz dats="I" mmez'spring mase scale 1/ coumeriz dats="I" mmer" spring lierris, seale" coumeric datam's' umer' maxieer__aaaa 15 cicaston>
cxarldholy>
<light diffesc=".5 -5" pos-"0 0 3" dir='o -175
Arena
<goo conaffiaity"* contype=""" natee minee"grondd pround pos-"0 size="l type-' pluse (gba="] 17>
typen"capuule")
conaffinily fromtoe'
'/>
egrom conaffiallya* ond" 1oomtot-33
names" sizea" 02" typeatcapulle" T>>
conaffially="0 otd"
fromins"-3
01 01* mames ddew" sizes 02" typestraponte Ts
di-- Arn
ggoo conaftiaity-* e'0' "" fremto="0 0 0 0.02" tames' root" sil"="001" typen copsatee"""" <tody tane- bodyn" pos="0 0 0.01">
<joizt ssise"l 0. 1" limited"'tree" mune""eizz0"" posn'd 6 8" synge" rangen"-3:13 8.13% <pesm frontos'0 0 D 0.2 0 8" mime""izko" sico=".01' typen'sapaule"
cbods nunee' fingeriip" " 205="0.11 6'>
spon nimes "iegerip" poeath 8" 01" rypes"spher"">>
clhndy
</undys
Target
<hody "trge" posa"0 0.01>>
dimping="0" limitzd='irue" tune""trzei_a" rasse="-22 stoffaess="0" stide amaauueen" 2 6" dumping=""" limitsé="'rue" tuge"""s"""""" rassc-" stiffaess="0 type= slide"")
conaffinity contspe="d" 'arget"
sphere
ethods>
d/yoridbudy>
<actuator>
emetor arllimiteda"Ino arlinnges-11 1.0" geara"200 D" joint17)
elactnator
diasjoco
E.3. Safety Critical Reacher
The Safety Critical Reacher environment builds upon the Reacher ID environment by introducing safety aspect. In 25% of the episodes, randomly positioned obstacle is introduced, which the arm must avoid. A contact force sensor is added to the observations, and penalty is applied to the reward for any contact force exceeding a threshold. This encourages the development of low-level controllers capable of reacting quickly to avoid collisions. The high-level state space adds a contact force sensor to the Reacher ID state, while action spaces for both high and low level controllers remain the same actions can lead to
as the Reacher ID environment. This environment directly tests the hypothesis that significantly better performance in safety-critical scenarios, aligning with the intuition presented in Proposition 2.1. The
24

---

EvoControl: Multi-Frequency Bi-Level Control for High-Freqqenny Continuous Control
faster reaction time allowed by high-frequency low-level controller is crucial for effective collision avoidance. To ensure reproducibility we provide the full environment MuloCo xml specification below. We also use the following reward:
π 90 2
3.1415927 I(IISel > 0)
(2)
where go is the joint angle (where go indicates the first dimension of q at time t), fc is the contact force between the arm and the obstacle, and I(-) is an indicator function that equals if the condition inside is true, and 0 otherwise. Where we used a fixed goal location of T/ 7/2, and initial starting state of qo = O. To ensure reproducibility we provide the full environment MuloCo xml specification below.
caujoco mndel="ssfenn rexches
<compiler anglew caddian" incriaftonoome"ree""
edefaelis
ojoint armatares"1" dampinge"I" lmmitdd"'mee"/>>
cprom friettonn' 0.1 0.1" zghan 0.4 0.33 0.26 1,0%
<ldefault>
coptics gravitya"o 9" timestep="" 002" />
crastom>
bra custon parsss
Koometic date=" 1.57 0.0 0.3" stitfness
opes
dats="1000
dats="1000"
danpsis commrii dats="..1" nomerconooraaa ang domping commeric dats="o.0" mimestnng damping")
<oomztic date="o" smic='sprits, mess_ 1/x <nometic date="]" umn--sppinng iiertiineeeeeeeeeee scale">>
<oometic data="5" amme= seoe"ssller___aae n>xis">> clenioms
<waeldbody>
clight d 3 3" posn70 y. dir="o -17/
ci-- Arena --
cyuum conaffinity "0 conispes"0" umez "round" 20x2"0 0 sizes't types' plane" rebaz") 1 17> <joon consffially contspe=""" freato""
-3 sidd=" size type=" "apssle "idde""
sgon conaffiially contspe=""" frentos"
types" "apeste"/s
typee'
conaffiaily="s ""0" contspe="9" fremto="..
tspee "apazle
syum consffinlty="" sta" contypes""" fromtnz00 0.0 D.D. 02" tomes"root" cile csprstees <tody ramen body0" puss'0 0 0.01">
sjoist axis="0 0. 1" limited='true"
uum=="oicath pos="0 0 9" 11550="-1.570 3 "ing=">
egeom fromtoa" 8 b 0.2 0 8" amea"lieko" sizeat.0n D1"" (ype='capull"'>
shody names" fingerilp" po 11 o 0's
<poon cousifisity 6-' contype= '0" amee"" fingertip" pos-"0 0 0* "ize=","1" upe="sphere"/>
<hody>
</hodys
Randon Collision Capriie
<body names' obstaccs -oay" pose"0 0.01">
<joint
demping=""" limiteda"tree"
x" pose"o rangeat-3
typea"stide" srmatorea")
<jaint axis-"t dompings"0" limited=" amme="obitell__. pos="0 type"'esssol mane"' 'osttdddbseed'>>>>
y" rangest-3
stiffoess="0 "slide
pos-"0
'02" fremte-"0
<lods>>
</worldbudy>
<costact>>
epair gremt pohhttoharscee" goona' fingertip" condimat1" is
<pair poohttsoos gonns line"" condim="1" 1>
olcostacts
<actuator>
cuettor atcllimited "tree" acdingg=--1100 1.0" gex:="200.D" Joiet">"s
clactuator>
<imsjoco
E.4. Safety Critical HalfCheetah
The Safety Critical HalfCheetah environment augments the standard HalfCheetah task with single, sertically-orientte capsule obstacle that the agent can collide with. At the start of each episode the obstacle is placed directly the cheetah's s trajectory with probability 25%; otherwise it is translated below the arena so that no contact can occur. The observation vector is extended by the scalar magnitude of the measured contact force, while the 6 6-DoF action space remains unchanged.
Whenever any the torso, fthigh, fshin or fooo geoms touches the obstacle, a reward of -200 is applied. We omit the usual quadratic control cost so that collision-avoidance is the dominant optimisation signal. The per-timestep reward therefore becomes
It+1
Ft.
200 m(l.fall > 0),
Tt
At
where x, is the torso's 's x-position, At is the physics timestep. fc is the contact force between the agent and the obstacle, and TT-) is the indicator function. As argued in Proposition 2, this large, impulse- shaped penalty rewards policies that can react high frequency avoid dangerous contacts. To ensure reproducibility we provide the full environment MuJoCo xml specification below.
cunjero model chrerah">
25

---

EyoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
coompiter ongles"radion" cocrdinate= bertiafrongeeme trae"" eetotallasseeee
cdsfaelt>
cjaint armatsre=" 1" dampingst BI" limiids='tre calingliatrz"" los colrefitmira" 02 <ilffeecss///
<aea consffiaity= ""0" condin="3" coatype="g" frictioz= a solimpp=00.0 0.8 0.01" solref=00.02 1" 12.a="0.4 0.25 1.0%
smetor stellinited="nne etrinaege=" 17%
"300000" ceptica
iterations="4"
d'-- brax
<nonttic date-"d 3.000 0 0 D 9 10" sunee iail, qpos"i>
<oometic datt="10000' (anc= constroint limit, stitfness T>
<oumetic da00="2000 tame=" constraint stifcness V>>
<numetic dats="10 22 22 2 1 1" names porstriii__ns_ddppiiie <nomiti date="10" uane=" constrainii_ell.e..... ve coumitic date=" 0.01" mane usg deg_dnn 1/x coummic dats="..2" amessbaugarte "tp" coummiic dats="o.3" names'spring_ mao_cacaal sometiic date="O.""" mmeessppring. inertia_sceee">>> commetic datz="50" solversolver maxis"/>
builicc"ggadient height="100" gall"") rabl="d 0" typee'skyoo 110th="100%>> <resture builitee" "lat" heights 128 makes cre markeghe"l I"
random= "0.01 01" rebit=to.8 0.6 0.4" r2 0.6 0.4" Ispenteabe" widts
15
<testere builline"
beight='100" umees tespiannn"" shi="0 o" r232="0. 0.8 0.8" type 22d" vidis="100"/>
<matorial names" MaiPlars" reffectince 2"0.5" chisleesss""" specutara a resropestaron on oo" tostareatrespianet
cmattrrill names" gron" tactures" innggonn ieciiforma"trar'
<lasset>
cwaildoddy>
<light sxtetf="100" diffose=") I" dir="-0 0 -1,3" directional= exposent="]" pos="0 0 1.3" specelar=" (1 177>>
cqun conaffinttys"' condine""" materiale "MiPlaan" sieee Hoorr" pose 0" sixes"40 40" sypen "plane" rgham 0.5 1,07s
<body sunc""oose' pax="0 0 .7's
posero
syatesell
ejaint armatinas""
domping="0"
pesert ciffressst" typostetide t ciffrens""" sifffees="* Sype"Thnnge '
cjaint antaturew"d" axis="0 hampin="00" limtize="false:
"0" limitzd="alsse rote""roty" unne"torse" size="C.066" type= "appe""eeeee
fromee '55
contspen" asisunglen"o 1 87 umeersaad
s.00 .15 tapen'csp
d-- <site nane tip pou- 115 0 .11/-->>>
<oody tance bibizi 5 9 0*>
qjaizt asise"l I 8" samping=' tibrih pore"l B 8" rangest-33 LOs" stiffnass""ui types"hinge's Ts
speom contypes""" scisangle="0 1.0 -3.8" nimes binigh pocerd -15" si2e5"0.045 .155" sypes'eapatee's
<hody umee bskin" pose".16 -25'>
<joint rsis="0 1 8* domplag= 4.5" mmme""biii"" poss o or conge="- .785" stiffness="189" "pp="hing''>>
spon asisingles'O 1 0 -2-07 canra "bshn" poss 14 0 -07" sizeato 11"" appe'"pp"ll">>>
<hody unees bloot" pox="-.88 -.1''s
<joiat axis="0
demping= amee" poln't cange=" .355* stiffaess- bisee" omes paxs"
compyres asteangles etbadys
<thlyy>
shody ramea rthighh poca".3 0
<joist sxis='d I of Sumpitg- '4.5" sunce flbigh" pos- 0" range="-] JJ .7' stiffness= type="ainge', Vy
egen costrgew"(" asisengle="0 1 .5" mame timee" pos"" .07 0 .12" size='0.046 .335" type- V>
<hods umee'
fahia" fahia" pos="..14 0 22'5
sjaint asis="l 1 I" tompinge's" names fchin" pore"u 6 8" rinen'-112 .77" stiffnews"120 typen"hine" Ts epon asismnzie' 0 1.0 -.5" mime""sshin pss 0 -09 size .105" tspen" capazle'ls
<body tleut" poa="_13 -.I8's
<jeint axiss")
dsmpinga"i amestfoon' poas'l of ragges-- 3" sriffansss"0rr hipg""i sgen contypra"1" asieangles -5" amestfoom" posst -03- exxe 2006 oor oxpeee/s
<ladyy
<lboly>
<holy?
Randon Collisioa Capsole
sames'ahstatla-a -hall por 0.01's
ejaint acix=70 1" damping="0 limited="inue" amee"abeaclee. pocet0 8" raage==10000 10.5" stiffsous"0 types" clide "s poxe"U 0" size="1." fromies 0 -5.0 o.0"... 5.0" types'appull" mimes" oberrcle <'body>
</worldbody>
scostact>
<pair "obitalles zcomz=' torse" condim="1" />
<pair geanl "obitalle" 1con2= fshia" condim="]" /s
<pair pobstele" gomis floo"" condim="1" 1s
<pair goomi oobstaall gomee' likigh" cuncims" />
<icontoci>>
<actuator>
geara"130"
hthighh ht bsknt="bskia"
"y bfoot"" sune"' bfoot"> joinre" fihigh" nimes" Hibigh emotor "ear="60" jshni""fshin" sames' fshia"/>
emotor gear="120"
emotor 3ear='30" jeints' ffoot" sunee' ffool"/s
slactuator
<lmsjoens
E.5. LoweLevel Controller (PD) Parameters
For every benchmark environment we release the exact proportional (Kp) and derivative (Ka) gains used to initialise the fixed PD c These values were obtained with the critically-damped heuristic described in Appendix E and are reproduced verbatim to enable reproducibility.
Position-Delta percentage. All position-delta controllers use a common relative percentage step size of 15% of the low-level action from the low-level policy multiplied by the joint range.
'Ienntical values are used for the initial Spp in EvoControl ($3); for the annealing schedule see Appendix G.
26

---

EvoControl: Multi-Frequenie Bi-Level Control for High-Prequency Continuous Control
Table 8: PD gains for locomotion tasks (joint order as in the simulator).
Enviromment
Kp
Ka
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,
[C. [0.013553226229071167]
1.011
C.019397106015317088,,,,
C.013502862280716777,
F.013397100015517088,,
C. G:013503286228076677,,
0. 0.013881006015517089,
C. G.013503286298071677,,
C. 0.013981006015517088]
HalfCheetah
[1.0, 1.0, 1.0, 1.0, 1.0, 1.00
10-0133558637744440077
C. 6.0120339779933553331,
G.0114991329420377111
G.011834333226533665
C.0152330080723322119,
G.022776070739666022]
Hopper
[1.0 O, 1.0, 1.01
10.027771661794882566,,
0.01664347577002351166,,
0. 0.010611222899772203377
Walker20)
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
[0.0433162629443355777
0.02100936632337/088.
c. 0.0051250366001966897,,
C. 0.0431162629953955774,
C. 0.0210093653220377088;
0. G.00512503360079888977
lumamood
[1.0 x 17]
[c. 10.0629446732887602222
0. C.C10-8849433275527233,
C. 0.055532203340077362,,
0. 1.0290122858820510166,
C. 0.003226277335880966600
c. S.009664388811402199,
c. 0.000334559458845555
CC0032227753880094440,
C 0.0096689838833555581,
C. C0060344509433899555
C.0291225530227586000,
C. 0.034117993125741108,
C. 9.019725189866474675.,
0. 0.0291295500225866005
c. 0.034117993155741099
C.C197251889564746771
Humanoid Stand-up
[1.0 x 171
10.05544770055777000022
0.05477339733333333199
c. 6.0143335546002229888
c. 0.00322262763330096622
0.C2901128538205105,,
C. 60096664888811402216,
80055692288334400099622
C. C.00222667635580965222
00290012285882051003
C. C.00966898388395655,,
c. 0.0056921803340019922
C. C.029129550002758832,,,
0. 0.0341179991225741108,
0. 0.01977211886644467555
C. 0.029112955002738869,
0_0341119993125744009,
C.C1972318966647467888
Inv. Double Pendulum [1.0]
[0.01737559333778611
Inverted Pendulum
[1.0]
10.078716111566329999
27

---

EvoControl: Multi- Fulqu-Frequency Bi-Level Control for High-Frequency Continuous Control
Table 9: PD gains for manipulation and reacher-style tasks.
Environment
Kp
Ka
Pusher'
[1.0]
10.009740730108445565
0. 0.0077384856482775398
0. 0.002149051382264499,
0. 0w00002236502199266002
0. 000000022999222999999
0. 0.00203307221327888000
Reacher (2-DoF)
[1.0]
[0.01, 0.01]
Reacher ID
[1.01]
[0.1] [0.1]
Safety- Critical Reacher [1.0]
F. Benchmark Method Implementation Details
Benchmark methods. We seek to compare against competitive established baselines, using the same high-Ievel PPO policy (p) learning algorithm with the same high-level architecture across all baselines, varying only the low-level policy (8). We compare with fixed controllers baselines, which are deterministic PD controllers of: PD Position, PD Position Delta, and PD Integrated Velocity (Aljalbout et al., 2024). We also compare against direct torque control baselines at both high (500Hz, i.e. the simulation timestep) and low (31.25Hz) frequencies; and a Random policy (31.25Hz). Moreover, we seek to investigate the EvoControl framework, and hence benchmark against several different variations from varying the observation for the low-level policy, from the full-state to a restricted partially observed state (only observing the robot joint positions g or velocities i), following EvoControl types with their corresponding observations as outlined in Table 2. Additionally, the EvoControl variants using position-based controllers annealed from their corresponding PD controllers. We provide more detailed implementation information for each benchmark method in the following.
F.1 High-Level Policy and PPO Implementation
The focus of the paper is enabling high-frequency control with learning based method, therefore to provide thorough competitive implementation of all the benchmark methods we use the same high-level policy neural network architecture and learning algorithm of PPO (Schulman et al., 2017) across all the benchmark methods for all the main results. We did also perform additional ablations of training the high-level policy with ES instead for all the benchmark methods, which can be seen in Appendix J.9.
We use the standard PPO implementation (Schulman et al., 2017), specifically using the implementation from PureJaxRL" (Lu et al., 2022), Jax (Bradbury et al., 2021) implementation of PPO. We used the fixed PPO hyper-parameters from PureJaxRL, which are derived from the PPO continuous-control environment parameters from CleanRL (Huang et al., 2022) which are themselves derived from those from stable baselines (Raffin et al., 2021). These hyper-parameters have been determined to provide good performance across a range of continuous-control environments. These param- eters are specifically 'learning_rate :3e4- 'num_nnmmn lareeeesss==l =1024, 'num_ntempp out),
(number of environment steps per roll- =1e6, 'update_epochs =4 (number of PPO update epochs per iteration). 'num_ caitimmmmiaathhhssre =8 (number of minibatches for each PPO update), gamma' =0.99 (discount factor), 'gae lambda'= =0.95 (generalized Advantage Estimation parameter), clip_eps' =0.2, =..2,
woc.cccef=000 'vf_ coa_ccoef===55 and 'max_grad gerad_nor''' =0.5 (gradient clipping threshold).
The PPO implementation uses batched environments for efficient data collection, accumulating transitions before performing updates. This facilitates parallel environment interaction and accelerated training.
x 'num_steps'
The high-level policy architecture (the same for all benchmark methods) Po (with parameters 8) is represented by an actor-critic network implemented using Flax (a Jax based neural network library). Both the actor and critic share a common base network consisting of two hidden layers with 256 units each and tanh activation's 's (this architecture was initially provided by PureJaxRL provide effecctve performance). The actor head outputs the parameters of multivariate Gaussian distribution (mean and diagonal covariance) as outlined in Section 3. The critic head outputs scalar value estimating the state-value function.
TThe Pur PPO implementation can found here htttpp://github.cco///ooue/ http://cccccccuuiiir//////rrrrr.
28

---

EvoControl: Multi-Frequency Bi-Level Control for High-Prequency Continuous Control
F.2. PD Controller Implementation
We implement standard PD controllers as described in Table 1. For all environments, we tune the PD gains described in Section E. Briefly, Kp is set to 1.0 and K2 is selected to correspond to MuJoCo 'dampratio' of 1.0 (critically damped). For the PD Position controller, the high-level action ag is interpreted as the desired absolute joint position (g"). The PD Delta Position controller interprets dj as change in joint position (ōq") relative to the joint position at the time of the high-level action Okx such that gd - + 5q" The Integrated Velocity controller interprets as the desired joint velocity and integrates to obtain a target position. This integration is performed numerically using the trapezoidal rule. These controllers provide variety of baseline behaviors for comparison.
F.3. Fixed Controllers
The fixed controllers (PD Position, PD Position Delta, PD Integrated Velocity and PD Position & Kp) are implemented as deterministic policies. Given a state and the high-level action dg. they directly compute the low-level control action Me based on the corresponding control law as described in Table Specifically, in PD Position & Kp. the high-level policy outputs both target position and Kp gain, which then used by fixed PD controller with fixed derivative gain-this allows the high-level policy to directly control the responsiveness of the low-level controller.
F.4. Direct Torque Control
For direct torque control, we use two variants: high-frequence (500Hz) and low-frequency (31.25Hz). the highefreyuency variant, the policy operates at the simulation frequency, outputting a torque command at every simulation step. The low-frequency variant operates at the same frequency as the high-level policy in the hierarchical setting. It outputs torque command every G = 16 simulation steps, which is held constant during the intervening steps. Both variants are trained using PPO with the same hyperparameters as the high-level policy, except for the number of environment steps which is adapted based on the direct torque control policy frequency. This allows for direct comparison of the performance of direct torque control at different frequencies. The same high-level PPO implementation is used to train both the high and low-frequency policies, ensuring that any performance differences due to the control frequency and not the learning algorithm itself.
29

---

EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
G. EvoControl Implementation Details
In the following we provide implementation details for EvoControl. We used JAX (Bradbury et al., 2021) to implement EvoControl, and present the core training loop in Algorithm 2.
Network Architectures. We the exact same high-level policy architecture and learning algorithm as the baselines use, from Appendix F.1. Therefore the high-level policy P9 (with parameters 0) is represented by actor-critic network implemented using Flax. Both the actor and critic share common base network consisting of two hidden layers with 256 units each and tanh activations. The actor head outputs the parameters of a multivariate Gaussian distribution (mean and diagonal covariance). The critic head outputs scalar value estimating the state-value function. The low-level policy S&NN N (with parameters ()) is separate neural network, also implemented using Flax. consists of three hidden layers with 256 units each and tanh activations. The output layer produces the low-level control actions (torques), T ue (unless otherwise specified, for example Kp, Ka). Specifically the low-level takes as an input observation the EvoControl variant observation, as detailed in Table 2.
Table 10: EvoControl Ablation PD Controllers.
Controller Variant
BNN Obs.
FNN action
EvoControl (Full State)
8c, Ogg ce: 9o Qr: L/T
EvoControl (Residual State)
ex, 1/T
EvoControl (Target + Proprioceptive)
ak qt, tt, Ct, t/T
EvoControl (Target)
0k q q1, t/TT
EvoControl (Learned Gains)
8c, uk Se, dr, 1/T St. age Ce Ut Ge, L/T
Kpp Ka
EvoControl (Delta Position)
Low-level Observation. For clarity we reproduce Table 2, here as Table 10. Specifically, the observation for the low-level policy can consist of the current state St, the high-level policy latent action ag, the PD controller error ez (that is used during the annealing), the robots generalized positions gr, the generalized velocities dt, and the ratio of the percentage of the low-level steps that the current high-level action is being followed for-for example with G = 16, T = G = 16, and hence t= (Algorithm 1) or the number of low-level steps out of G that the low-level policy is currently on whilst following the high-level policy.
Annealing Strategy. The annealing parameter a controls the convex combination of the fixed PD controller (spp) and the learned low-level policy (9øNN). We use a linear decay schedule, starting at a 1.0 and decreasing linearly to a 0.0 over the K training sections, of ag
& where is the current training section.
ES Details. For ES we use Policy Gradients with Parameter-Based Exploration (PGPH) (Sehake et al., 2010) algorithm to optimize the low-level policy SONN. The neural network's parameter vector ő is directly optimized. We use a population size of es eee_ee__psi___ieeeeeeee pop_size- = 512, and each individual is evaluated over es rollouts = 16 rollouts to estimate its fitness (episodic return R2.. Adam (Kingma Ba, 2014) is used within PGPE, and we we use the PGPE hyper-parameters a center learning rate of 0.05 and standard deviation learning rate of 0. We use es_sub generationns 8 generations per training section k. The parameter distribution's 's initial standard deviation is 0.1. We use the implementation of PGPE provided by EvoJax (Tang et al., 2022), in Jax, and their recommended dyper-parameters for PGPE, which were empirically found to work well for continuous control tasks. Furthermore, we set K 8 per 1M high-level p steps used to train the high-level policy for, and this was empirically determined to work well in practice.
G.1. EvoControl PseudoCode
For the following pseudocode; we used the same parameters as described above, specifically setting the total number of training sections to K = 8 (per IM high-level policy steps), and then training the PPO high-level policy for 1M (i.c. I Million) high-level steps, therefore performing N PPO updates of N = 1e6/(num envs nemm_ x [x]] = 12. We note that each step of the slow high-level policy when operating with a fast low-level policy is effectively G = 16 high fighffrequency environment timesteps of A,.
30

---

EyoControl: Multi-Frequeney Bi-Level Control for High-Prequency Continuous Control
Algorithm 2 EvoControl Training
Require:
f(s, ur), reward function r(st, uz), high-level policy po(s)), initial low-level policy SPD(S ag). total training sections K, steps per section N, annealing strategy for a, ES parameters 7/, population size P. generations per section Gevor rollouts per individual Revo-
Environment
Ensure: Trained high-level policy (o(8s), trained low-level policy 30 (8;, (x))
1: Initialize a - 1.0
2: Initialize low-level policy [ss;, az) < a a&) + (1 - a)) (k)
3: Initialize ES strategy (e.g., PGPE) with parameters y
4: for k
K do 5: // Train high-level policy pe with PPO
6: for 1 to N do
7: Collect rollout data using Po and 3 (Algorithm 1)
8: Update Po using PPO
9: end for
10: // Train low-level policy B& with ES
for g = Gevo do
Ppop Sample P parameter from Py (o)
13:
for p 1 to P do
14:
Fp 0
15:
for r = 1 to Revo do
16:
Collect rollout using (mode) and 3a, (Algorithm 1)
17:
Fp f F-t rollout return
end for Fg + -Fp/Reue
20:
end for
21:
Update ES parameters 7 using fitness values FLP (e.g., PGPE update)
22:
end for
23:
& t- best performing parameter set from ES
24: BaNN(8i, ax) t neural network with parameters &
25:
A(si, ax) f- a:BpDss, 0x) + (1 3))
a x k/K
27: end for
G.2. Detailed Analysis of EvoControl
This appendix provides a detailed analysis of the EvoControl algorithm, addressing the mathematical setting, assumptions, complexity, and properties as requested.
G.2.1. MATHEMATICAL SETTING AND ASSUMPTIONS
EvoControl operates within the standard continuous control Reinforcement Learning (RL) framework. We consider Markov Decision Process (MDP) defined by the tuple M = (S,U, P.r,)), with definitions provided in Appendix A.
Assumptions:
Markov Property: The environment dynamics satisfy the Markov property, meaning the next state depends only on the current state and action, not on the history.
Stationarity: The transition probabilities and reward function are stationary (do not change over time).
Differentiable Policy: The high-level policy (s(s)) is parameterized by 0 and is differentiable with respect to 6. This allows for gradient-based optimization.
Representable Low-Level Policy: The low-level policy (st, ax) can be adequately represented by the chosen neural network architecture with parameters a
31

---

EvoControl: Multi- Bi-Level Control for High-Frequency Continuous Control
G.2.2. COMPLEXITY ANALYSIS
Time Complexity: The time complexity of EvoControl is dominated by the PPO updates for the high-level policy (p) and the rollout evaluations for ES of the low-level policy (3).
The per-update complexity of PPO scales linearly with the number of environment interactions. For No high-level steps, with 'num_envs' parallel environments running for "num_stepp steps each, and PPO updates occurring every K training sections, there are N num_envs num_steps) PPO updates. Each PPO high-level environment step involves G low-level environment steps.
PPO Updates:
_s__o___s________p______ Each training section involves ES of the low-level policy. With a population size of 'es _op__izee 'es rollouts' rollouts per individual, and 'es ss_sub__rt uua__eeerattoon' generations per section, the number of rollouts per section is es os_roolouts es sub suuu_gennrattinn cs_pop_size Each rollout has "episode_length low-level steps.
ES Rollouts:
Let Teav be the time for single lew-level environment step of duration A Then, the total time complexity, without parallelization of environment rollouts, is:
O ((NN- G + K es es_roolouts- es_sub seuuubggerttionss cs_pop_sizze size episode_ length)
Tenv)
We can also re-express this, as we train the high-level policy for N, high-level p steps, and we train EvoControl with K sections per IM high-level o steps (i.e. K (N, ()/(1)))), then the time complexity can also be expressed as:
O(NN, G+ No
es ees_rollouts es cs_sub____________ generatttsss es ss__oo__staee size episode _pisode_length)
Tewv)
le6
However, both of these time complexity measures are worst case, and do not account for any availability to parallelize environment rollouts, which is common in practice on modern GPUs. If we assume that a user has a GPU/CPU that can parallelize the environment rollouts, then the time complexity can approach:
O(Nm num steps- G+
No 8
1e6 ess_ ees_sub se._rb__enertttons pepisodelengthhe Tenv)
Where Nn = (N,))(num_enn num num_steps).
We provide thorough additional experiments limiting the computational complexity of the above two approaches, as detailed in Appendix J.4.
Space Complexity:
The space complexity is primarily determined by the size of the neural networks for the high-level and
low-level policies, the size of the PPO buffer, and the population size for ES. It is 0(10) + lø + Sppo + P- (øl). where (@) and |o] are the number of parameters in the high-level and low-level policies respectively, and Sppo is the size of the PPO buffer.
G.3. Computational Considerations
Building on the previous section, environment rollouts can be parallelized on modern GPUs. Specifically num_envs, es sesrrollouts, es pop_ssi_eeeeeeee__e_ee__eee can all be parallelized. A benefit of ES here, is that the fitness evaluations (within PGPE) are highly parallelizable. We leverage JAX's 'vmap' function for vectorized rollouts, enabling efficient parallel execution on GPUs or CPUs. This can also be readily further optimized, such as distributing the population across multiple devices, to reduce training time (as ES is gradient free approach) (Salimans et al., 2017).
Whilst the goal of our work is to provide an initial method that can learn better low-level controller for use within high-level policy learning environment, to achieve higher final evaluation reward, we acknowledge that doing so increases computational complexity compared to policy learning with a traditional fixed PD controller. Therefore to investigate, what happens if we make the number of low-level environment steps equivalent we provide a further ablation in this setting in Appendix J.4.
32

---

EvoControl: Multi-Frequente Bi-Level Control for High-Frequency Continuous Control
H. Evaluation Metrics
For each environment, and for each baseline we train the joint policy т consisting of a high-level p and a low-level policy for the same number of high-level policy p steps, here 1M steps. Once the policies have been trained, we perform 128 evaluation rollouts, each with a different random seed and compute the undiscounted cumulative sum of rewards for each rollout, i.e. the return for the episode, where each episode lasts 1,000 environment steps. We repeat training each baseline policy across three random seeds. We quote each result as the mean across it's random seeds and provide the corresponding 95% confidence intervals throughout for all metrics. Specifically we quote the normalized score R (Yu et al., 2020) of the policy in the environment, averaged over 384 random seeds- to the interval of O to 100, where a score of O corresponds to a random policy performance, and 100 to an existing fixed controller expert policy- which is whichever non-EvoControl baseline scores the highest evaluation environment return.
All experiments were run on NVIDIA H100 GPU, with 80GB VRAM with a 40 core CPU with 256 RAM. We detail the were selected, and their origin
in Appendix F for each benchmark method, and how the of source. We did sweep over the learning rate for PPO with the fixed controller PD position baseline, however found the initial typer-parametert already provided by prior work (PureJaxRL (Lu et al., 2022), and hence CleanRL (Huang et al., 2022)) to be the most performant, therefore they were kept constant throughout all experiments.
I. Additional Experimental Setup
L.I. Efficient Exploration Experimental Setup
To reproduce this experiment, we used the Reacher ID environment, as detailed in Appendix E.2. Specifically to investigate the efficiency of exploration, we modified the Reacher ID environment to have deterministic goal across new random seeds, such that the goal location is Aguul = #/.00, and the initial starting position to 0. This is to ensure that we can correctly measure exploration, otherwise starting in random state with a random goal, could already explore the state-action space, just through environment resets-whereat the focus of this insight experiment is to compare the methods exploration instead, hence fixing the environment starting state and end goal state. Specifically we run each baseline approach for 10,240 levelevel environment steps each, and collect the state throughout training for these initial steps. We then process the state collected, and plot the state visitation histograms, as shown in Figure 2.
1.2. High-frequency Interaction Control in Safety Critical Reacher
To reproduce this experiment, we follow the same setup for the Safety Critical Reacher environment, as detailed in Appendix E.3.
J. Additional Experiments
J.1. EvoControl Supports Other Competitive ES Approaches
EvoControl supports using other competitive ES approaches to optimize the low-level policy neural network parameters. We provide full results for diverse competitive ES approaches, where for each ES approach we replicate our full main-table of results presented in the main paper. We provide this analysis empirically, as in Table 11, then follow with a discussion about which ES user can select use within the EvoControl framework.
We benchmarked against the following ES approaches of:
PGPE (Policy Gradients with Parameter-Bate Based Exploration (Sehnke et al., 2010)): policy search algorithm that optimizes neural network parameters by maintaining a probability distribution over them and estimating gradients directly in parameter space. Unlike traditional policy gradient methods, which suffer from high variance due to per-timestep action sampling, PGPE samples policy parameters once per episode, ensuring deterministic rollouts within each trajectory. This leads to significantly reduced gradient variance and improved learning stability, particularly in reinforcement learning tasks with long time horizons and continuous action spaces. PGPE also supports symmetric sampling. which further refines gradient estimates by leveraging paired perturbations, akin to central difference approximations in finite difference methods. These properties make PGPE particularly effective for optimizing neural network policies in complex, high-dimensional control problems.
33

---

EvoControl: Multi- Fretu-Frr Bi-Level Control for High-FFrqqenny Continuous Control
Table 11: EvoControl benchmarked with other ES approaches. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 0-100 scale, where O represents random policy, and 100 represents the highest reward achieved by a non-EvoControl baseline in each environment. Scores bolded are greater than 100.
Sanee PPO high-leee da. P with Lowo-Lovel Pollcy Bol
Hallcheeuh
Hopper
Hammaood
Hamanod Standop
Inversed Doubt: Pendslmm
Invered Pandslur
Pusker
Reacher
Reacher
Wi
ID
R21
R
R1
R2.1
1
R1
R
Ploed Cost. Position
100106.50
M1.061.23
2000
100
999440.5
85.247.87
75/7-00155
Road Cost. -PD Proition Delns
2.:1
2.76:000655
1001-135
96.111.71
2 296:006077
51.8:1.57
100-2 95:-15
0::0
40,9::3:3
15:2:766
50:2:02219
Plaed Cost. Itr, Velocity
3.7
76.7609993
83.663.13
U::0
49.751.55
M.5-2
0::0
0:40
010
25.5
Findd Cost. Randon
0.0:0000
00:0000
00000.0
0.0:000
0.0:0.0
0,00000
0.0:0.0
0.0:000
0.000.0
0.010.0
0.0:0.0
Direet Torgur Cast -High Frey. (500H)
0::0
17:2:0336
142::0.533
10.41:211
50:3:0.566
0:10
0::0
1.34:5.89
2.0::5884
45 65.7
0::0
Drnet Toqqe
Frot. (31,25100
54.57.15
98:2.55
$0.622.56
100:0:0311
100
59.263.72
100
200:2:6
PGPE- -SvoCoattel State)
368
276:11.52
1231277
116:10409
1001-2.00-11
134:00.933
PGPE- EvoCuntte (Reidual State)
182 8.50
182 1.02
1,15
170:1,14
212 4.95
00:2100004
100:2.96:-11 15
375 10.7
106 1.69
104:1.19
208 2.72
PGPE- IsvoCoatrol (larged Progrioo.
319
108:1.40
171::5288
1.5:1.77
165
9477000177
1001 15
353-
96 543:355
10576
178
PGPE- EveCuatedl (Target)
293:
162-1158
283:1191
16411.
506
996600037
10082.962: 15
112:0.785
PGPE- EroCunttea (Laarned Caire))
266: 14.1
113:16
106:10
150:2255
117:00.06
50.5:000477
100:2:9:-15
330 10.5
1.6:0.747
185:1.21
1961:45
PGPE- EvoCuntrol DDllaa Position)
133-1182
2251:2.9
1192277
105:0.265
101:000364
10062.86:11
267:116
65.523.71
99.112.44
CMA-1S- EveCCantrol State)
104-159
43.5-12
10010.11
100:0
109:2.34
106:344
CMA-ES- EvoCeetted (Residual State)
37.2::29
102:1002
201::201
52:13.9
58/8:0162
1.040.224
100:40
259
112:4421
10613387
215:1113
CMA-LS- ErooCentro (largee+ Progeto.)
02.02:0.
595:16.4
261:1116
7474154
99774494
99.940.001
100:00 1000000000000
31N:41.4
215
186:36
201 201-2.56
CMA-ES- EveCoutaed (Target)
72.3450.7
98.5=12.6
212:2-07
76.977.3
85 $5.9
09.1000111
10040 100000000
307:5001
115455.33
10624
309:1111
CMA-ES- EvoConird (Learne Goins)
42.75 23.33
14.4
156:11.5
16.2
116:00379
006:106609
100:10 00::00000
319 43.6
111:541
183:5.00
166 17.2
CMA-ES- EvoCented (Dela Position)
31.6252.7
$5.4:941
1281:129
97.8
304
100:0.225
100:0
304:548
70
84.5120.7
135
SES-UMA-ES ExoUCntrd Statel
48.86613
15:8-176
108:10995
71.54135
10:00.57
100:0.126
10240
281
186:367
150:6-6
SEP-CMA -ES- ExoConteed (Residual State)
91.84759
138::415
246:233
122:133
108:206
99.6-00116
100:00
30.4
112:2.78
1061:425
271:2226
SEP-UMA-ES ExoUantre (Target. Progete)
52.36-37.
96.N-7M.
126-10915
546:220
307
994000009
100400
2W3-543
115:231
106:374
198:
SEP-CMA-ES EveCantred (Turac))
96772516
92.6:2218
12::00
53:21.7
104:417
987200031
10040
112:5.76
10G3366
19624
SEP-CMA-ES- - ExuCanire (Iramed Gains)
936:5515
73:27:00
15912252
107:11.9
116 126
100:0.137
100:10
320 43.8
11415.71
10416.3
151-13.6
Std-CMA-ES ExoUGntre (Detta Posttion)
44:8543
125--179
81.54833
100:50
900-009877
100400
85.5420
136:1155
CMA-1S-JAX- EveCCnntral State)
96:3-1622
113:10996
70.1455
100
100400
352--91
113-332
186:4,M
222:10.1
CMA-ES-JAX EveCostrel (Reidund State)
161-70.7
99.813383
14815522
114::6.7
98.5-0996
1.05
100:40
237:5541
111::667
10624.1
183:5525
CMA-LS-JAX EvoCContral (Target Propriso)
5656-00 49.3
17.7
128:3357
1655148
961-556
10040 1000000000000000000000
312:5777
112:3312
186:M8
775
CMA-ES-JAX EveCuatrel (Target)
10:115.2
126:11.36
61.7111
9..222077
901:0000588
10040 100000000000
40.3
110:3.44
10613.8
1.2:2.42
CMA-ES-JAX- EvoCuntral (Laorned Guire)
63.8:40.1
72:8:1033
1874
134:145
110 300
0.010.101
100:10
352
43.4
105 105:9981
186144.6
136 11.4
CMA-IS-JAX EvoCCnntral (Delia Postiion
67:
15:112.
107211122
64.3110
90.4-0215
99940.12
100:40
317:422
67.281552
87.142577
State)
335
5.96:742
32.460306
78
244:995
102:1055
74.24127
0::0
DE EvoContel (Residual Stute)
7.54
23.3
142:111
1.71
45.7218.2
01:0
235
107.7.37
1061:387
1.37
DE ExoContral
Proprio)
18.6:34
32:92207
99.7616.5
140
2977-125
491
57,3
196:392
DE. ExoCuntrel (Tarae))
0::0
13:3::618
20.9:6
2:35:00652
000-00066
92.1-3002
319:4478
64.8:17555
10615.11
0:00
DR. ExCCutrel (Lizared Gains)
0:0
22:1:5.32
00216775
D10
22 5:000
52:2233
100:10
220 40.7
53.7:1344
50.1:115
1.062276
DK ExoCintral (Delin Postion)
46,2410
41346000
10::22
27N 41.7
54.58158
43.71M
JAMaLGGM- Controt (Full
04.62114
75:24300
92240121
109:158
94.5-135
100-100011
100400
55.5-44.5
106:4411
iAMaLGaM EvoControl (Residual Stute)
110::003
101-9938
58880524
99.41144
96.2200953
99.410.162
100:10
333-552.5
109:3349
10613.6
161
iAMaLGaM-
EvaCuntral (Tangel+ Proprin))
54.3:55.
$7.6:166
110:124
105
004410.117
10030 00000000000000000000000000000000
327:44
63::1911
186:382
140:1122 18.2
1AMaLGaM-
EveCunteel (Targe))
8.0
1405571
9071100
98. 56.56
10010.144
10040
59.6:17.9
101:2115
iAMAL.GMM-
EvorCuitel (1 (Lazaned Gaine)
5.99140.1
53.6:581
12518/91
11.7
111 1.82
0.000
100:10
266 50.6
114:8A2
102:591
958-874
JAMaLGaM
ExvoContrll (Delin Podtion)
13 49
78665719
1211-132
10219.36
10530
300
54.4-194
104:109
1.6-7.33
CR-FM-NES EvoCCnntra State)
85.54 15.565561
71:
962400125
06M1133
100400
351
55.5--4.5
106:441
123-444
CR-FM-NES- EvoCostrol (Reidual Stute)
45.9:5112
134:133
588850544
129
98.423..6
99.410.162
100:10
32N 45.8
111:4.54
10613.6
184:7.76
CR-FM-NES- EroCuntrel (Target + Projrio.)
61 65.2
11::15
03:0295
97.11189
106 1.6
00.410.117
100:0
327 40
63:-1911
186:382
110:114 11.
CR-FM-NES EsoCuatrel (Target)
67:
72:2:122
1403571
113113
1.8:3.46
100:00144
10040
99.61179
1072400
113:411
CR-FM-NES- SeoCunntrl (Lacarned Calred
45.0167
56.7:6944
12518:98
120:143
979-3.2
1.0:0
100:10
313 44.3
1141538
102:
106 6.75
CR- CR-FM-NIS InvoControl (Delin Foottion
06:4:147
121--332
10.3
45
102:9.36
10030
300
54.51194
95.0131.1
OpentS InvoCottol (F'll)
104
2676:02055
08243.3
61720031
43.14324
100400
204:4431
113:6:44
OpmES EvoCustrol (Residual State)
15.62288
46885583
2110
115:
645500194
0:10
10::10
206:145
1141446
1031:633
0:00
OpinES OpenES-EvaCune EveCCuirol (Tanpet+ Propricd
3634-528
30.7:8.3
87.260.14 144
67:3.14
5122-0335
0:0
100:00
214
11327
105158
29
OpenES EvuCCateol (Target)
20.7663
0.:0
1350,0009
42.122.17
59:6:0:61
0:10
100:00
203:4776
85.1-99
3577242
10
Open8S EEsoCuntro (1 (Laurned Gaino
4390:534
181::15
$4.910.00 104
44444.4
02 7:5.6565
100:10
229 48.2
194:132
45.5:154
31.31666
Opent8 InvoControl (Delin Podtton)
6:22: 888
9.26501156
76743355
T34:00 102
72,95552
100:0
220:-005
565:201
910
11.7:991
PGPS- Diver Torgue Cost. High Ving (500H2)
306-311
14:7755
138:-482
954416.6
5855535
100:11.34
100:00
300:31.3
57.5:12.1
KM6:14
126:366
CML-ES- Divec Torque Cont. High Fice, (500Hz)
70.4120.1
5006:5653
118::3488
100
95.621.33
99.721.17
100:00
268::27
55:118
D10
111:274
SEP-CMA-ES- Diver Tongu Cant. High Fray (500H))
50.6:308
96:4:
102:9.51
K10:K82
35.3:330
84.71:996
100:00
213-1229
55.::118
010
110:385
DE- Divert Roque Cost. High Fres, (50084
57.32115
11415535
62.517.57
79.424.41
43,458.48
$7.9217.5
258:26.9
57.2211.
20.9110.8
$7.9:116
GaM Diver Tempe Cont. - High Prog, (800tis)
82.8:99
$7.7:10.1
77:4.55
7.57.56
90.9-136
83.91467
100:10
266-
59.611188
D10
114-7.32
Opents Diver Toruu Cont High breq (500H))
17:9
126:174
41.864.51
42 10:4
24.151.78
6.8662.34
112:355
25.1216
050
195:115
CMA-ES (Covariance Matrix Adaptation Evolution Strategy (Hansen & Ostermeier, 2001)): A widely used, competitive ES algorithm known for its efficiency in sigh-iimensional continuous optimization. It adapts the covariance matrix of the search distribution, guiding the exploration towards promising directions in the parameter space.
exploration
SEP-CMA-ES (Separable CMA-ES (Ros & Hansen, 2008)): A variant of CMA-ES that utilizes a separable covariance matrix, reducing the computational complexity for high-dimensional problems.
DE (Differential Evolution (Storn& & Price, 1997)): Another popular ES algorithm that relies on vector differences
34

---

EvoControl: Multi-Frequenge Bi-Level Control for High-Frequency Continuous Control
between population members to generate new candidate solutions. It known for its robustness and ability to handle complex, multimodal objective functions.
LAMaLGaM (incremental AMaL.GaM (Bosman et al., 2013)): A -AX-based, parallelized variant of AMal that builds its probabilistic model incrementally for increased sample efficiency.
CR-FM-NES (Cross-Entropy Fitness Model based Natural Evolution Strategies (Nomura & Ono, 2022)): An ES algorithm that builds a surrogate fitness model to improve the sample efficiency of the search process.
OpenES (OpenAI Evolution Strategies (Salimans et al., 2017)): A simple, parallelizable ES algorithm that utilizes isotropic Gaussian mutations for exploration.
When evaluating these different ES approaches within EvoControl used the open source implmentations from EvoJax (Tang et al., 2022) and EvoSax (Lange, 2023). Specifically using their respective defined implementations, and using the same broad ES pepeerppraamters across all implementations as defined in Appendix G.
We empirically observe in Table 11 that EvoControl can support competitive ES approaches. Specifically PGPE achieves competitive performance within the EvoControl framework across all of the evaluated environments. Interestingly, within the EvoJax ES library PGPE achieves the highest evaluation reward in the environments that EvoJax benchmarked against (Tang et al., 2022), which correlates with our empirical findings.
In practice EvoControl can be used with other ES approaches, however select PGPE for the main table of results presented in the main paper, due to its good emprical performance (Table 11). Additioanlly, PGPE is computationally efficient and well-suited for large-scale, sigh--imensional optimization problems typical in policy search for reinforcement learning. PGPE is straightforward to implementat and scales across multiple compute nodes, leveraging modern accelerators. Furthermore, PGPE has demonstrated robust performance in tasks requiring fine-grained control, making it suitable choice for optimizing and DE offer sophisticated search mechanisms, they come with increased computational complexity, such as: CMA-ES: Adapts the covariance matrix but becomes computationally intensive in high-dimensional parameter spaces due to large covariance matrices; DE: Effective in continuous optimization but may struggle with the noisy and dynamic nature of reinforcement learning environments: Advanced ES Algorithms: Methods like iAMaLGaM and CR-FM-NES introduce additional computational overhead without consistently yielding better performance in our context. Our experiments show that while complex ES algorithms can perform well in certain tasks, PGPE offers a favorable balance between performance, computational cost, and implementation within the EvoControl framework.
frequecyy low-level policies. Interestingly, more complicated ES algorithms like CMA-ES
J.2. A Ablation: Using PPO to Train the Lower-level Policy
We performed an additional ablation experiment, by training the low-level policy with PPO rather than ES, To be comparable we used the same architecture that our existing high-level PPO agent uses, as described in Section 3, and Appendix G. We follow the same setup, of training the high-level policy for 1M high-level environment steps, and now train the low-level policy for the same 1M high-level steps, now training for the low-level for 1MxG = =16M low-level environment steps-to give this ablation the most competitive performance comparison to EvoControl and the non-EvoControl baselines. We perform a complete re-run across all environments as presented in the main paper main results table. The ablation with PPO training the lower-level policy can be seen in Table 12. We observe that using PPO to train the lower-level policy within this EvoControl ablation performs worse (achieves a lower average evaluation return) than using ES to train the lower-level policy thus justifying the use of Neurevolution for training the low-level policy.
J.3. EvoControl Outperforms Direct Torque Control at High-frequency
In the following we provide empirical evidence for EvoControl outperforming the baseline of a high-frequency low-level direct torque control policy. To address any sample complexity concerns, we also find when we limit EvoControl to use the same computational complexity as all baselines, EvoControl still outperforms the baselines, which is evaluated in detail in Appendíx J.4. To provide thorough analysis of the ability learn a high-frequency low-level direct torque control policy, we performed additional experiments of training the Direct Torque Cont. High Freq. (500llz) baseline for an
'hhe ES approach benchmark results for EvoJax are provided here: https: pestps:/ggithae com cow/goooloeeoooooooooooooooooooo h/woo//////////eeeeeeeeo/a scri pt. https//eehhharaaeeee rks.
35

---

EvoControl: Multi- Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
Table 12: Ablation. Training the lower-level policy with PPO instead of ES, training both the high-level policy and the low-level policy for IM high-level environment steps cach, to produce competitive ablation. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 0-100 scale, where 0 represents a random policy, and 100 represents the highest reward achieved by non-EvoControl baseline in each environment. Scores bolded are greater than 100.
Mothod Nare
Htubllax pwil
Low-lord J' wil
Asz
Hallchonzh
Bagpar
Hamazeid
Hamased Sunska
Invated Doutae
Inseted
Pater
Headaer
Resster
WalkerDD
Poslalan
Peoéelan
ID
X1
R.
X1
X-1
X
X.1
R.1
X.1
12:1
Pbat Cost 111 Poerse
1911
PD Ivittion
100:1.:55
61.. 6.41
91611:23
2.0
110/3447
111111
25:212.87
757 75-710611
Find Cast PD Postion Deta
PPO
PD Pullisn Deha
24+1151
27660 0688
100s
296600000
53881157
100+1 1..55 0G
0:0
40/9:3:23
1522:76
50.260.299
Finnd Gost br. Volneity
PP0
PD bt Voloety
1.99-177 76
2:6600.00 9930
747401905
5344-113
697741 55
86:5-3
0:0
040
8599-2555
Tonge
Dirett Toqur Cenl. High Free. (508L)
PPO
Diect Toque
060
1722:0316
142440553
10.3605556
0::0
3.00:584
Divet Trgger Cos- Lowe Fray (5) 2502
PRO
Direct Taque
54.5-715
100-1:21
7240064
08:2255
$066-2.56
100:0:0011
100+1
73:2-12:
59.243772
10081
1000:66
Ablation: Intrral Witl State)
1910
1641.99
25.51336
182.5553
142: 2135
K211788
$1.51-0
69A1112
385 219
55.127
1011131
S2A1MM
Ablailen: EvoCantro (Roaktul State)
PP0
190
125-9.7
41:32892
0844100
0012:111
319:100
$461645
10020
94.72351
91A2112
101177
103:9822
Ablatlon: Fvost bantnel TTange- Propeie.)
P91)
IP0
040
35:5:336
54.1-724
465--443
16.5-574
St 6161135
39.6--333
91.3-153
4471-443
67 0358851
205-119
Abtation: anotaa (Target)
191
191
010
1738:167
18.
6514916
1551119
2781121
146
95.11:35
1551525
95:114122
Ablatiss: EveCantra (Ddit
increasing number of high-level P policy steps. Specifically, as tabulated in Table 13, we train for a larger number of p steps, significantly greater than all the baselines were trained for (which 1M p steps) here being from 1M p steps to 10B p steps. We observe that even with more high-level p steps, which corresponds to significantly more low-level environment steps than that used in EvoControl, Direct Torque Cont. -High Freq. (500Hzz) still on average achieves lower normalized return compared to EvoControl. This could suggest that direct high-frequency control with PPO produces policies that get stuck in local minima, and fail to find a better performing global policy at high-frequency as EvoControl is able to do-leverggin ES for learning the lower-level high-fre policy.
Table 13: Additional Experiment. Training Direct Torque Cont. - High Freq. (500Hz) baseline for an increased number of high-level p policy steps-from from IM steps to 10B p steps. Normalized evaluation return R for the baseline, across each environment. Results averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to 0-100 scale, where O represents a random policy, and 100 represents the highest reward achieved by a non-EvoControl baseline in each environment using the normalization from the main table of results, Table 3.
Same high--iett elp o Low Level Policy of
Hallcheetah
Hopper
Hummoid
Homonold Stardup R1
Inveried Prodalin R2.1
Roather
Pesdulus R
ID
I2.1
1
R1
g
72.1
72.1
1
1,0000000 Tran y stops Direct Torger Cint.
(500H2)
17.25:0336
363360299
156:3364
17=122
0:50
0:40
28557781
45.366774
10,000,000 Trais sieus Dover Evuue Cost. High Frot. (5006t2)
631:14.
63:1143
90
67.923.41
78.322
97:9:0.148
42.621.48 4B
21,919,03
42.455.9
57.8:3.62
36.5:1088
100,0000000 Trin a stope Dineet Rongus Cont. - High Fray (5008))
159:17
1061:144
13419.53
133
13913.5
88:2:108
63:119.6
42:5113.9
108:2.32
5761633
91.11..11
1,000/0000000 Train steps Diiect Torque Cost. Hiih (50010
125815.4
1011:1.55
14152.49 49
12753,79
119:300
17.511
86.811344
2192:111
9836-255
73.56604
18926-47
10000,003,000 Trais pattoo Direct Zergae Cost. High Freq. (5008t))
18581166
86.361.09
155
143:6.98
7133113
91.8:1000
94.621.21
75.56127
88.523.01
51.2:4.56
219:5.33
J.4. Ablation: Equal Computational Complexity for All Baselines
We investigate making the computational complexity the same for all baselines and EvoControl, in two approaches. First, the most direct approach we set the budget of the number of low-level environment steps to be equivalent for all baselines, listed as equivalent number of low-level environment steps in Appendix J.4.1. Second, we recognize that modern GPUs allow for environment parallelization. Thus, we investigate only setting the same number of sequential low-level environment steps to be equivalent for all baselines-whee the bottleneck for parallelized rollouts is the number of sequential steps of the parallelized environments. This is listed as equivalent number of sequential low-level environment steps in Appendix J.4.2.
J. EQUIVALENT NUMBER OF LOW-LEVEL ENVIRONMENT STEPS
Here we explicitly set the total number of low-level environment steps for all the baselines to be the same. For EvoControl, that trains its high-level policy with PPO and its low-level policy with ES, this means that the high-level policy trained with PPO now receives less high-level update steps compared the baselines, to account for the low-level step budget used by the low-level ES component. This is different from the main results within the paper (Section 5.1) which trained each baseline for MM high-level steps, thus meaning that the high-level policy was trained for IM steps, not accounting for the potentially differing number of low-level steps used by updating or using the lower-level policy.
To set the total number of low-level environment steps for all the baselines to be the same, we first compute the total number of lewelevel steps that EvoControl uses, where we train EvoControl's high-level policy for 1M steps, and then now train the baseline methods for this increased number of equivalent high-level steps. Specifically if we consider a PD
36

---

EvoControl: Multi-Frequenge Bi-Level Control for High-Freqqenny Continuous Control
position baseline, originally training this for 1M high-level environment steps, with a lower-level PD position controller, operating at higher frequency with G 16, meaning that the number of low-level environment steps used in the environment are 1M x 16 16M. Here EvoControl, when the high-level is trained for 1M steps, the lower-level policy also receives updates, therefore the total number of low-level environment steps used within the training of EvoControl is K x resrrollo es_rollouts x es_ -uuuuu__neratiionss x es_pop_size x ppiisdd_lengthe To simplify the comparison, we explicitly set es "s_rollouts- = 1 and leave the other inputs the same as they were for the main results (that of K = 8, es_sub guaee_ggerttionss = 8,
= 1000). This leaves the input parameter of es es_eep__spppp piiesize that we can vary. Therefore the total number of low-level environment steps used by EvoControl is 1M x 16 + 64K x es Sos_poo_siiie__ size. Therefore, for the following experiments we train all the other non-EvoControl baselines for 1M x 16 + 64K x es__op_ size low-level environment steps, by specifically determining how many high-level steps this is by dividing by G' and using that as the input as the total number of high-level steps to train for each baseline.
For the results, as discussed, we vary es os_pop_size = {16, 32, 64, 128, 256} and re-run each con-EvoControl baseline with the equivalent number of low-level steps as EvoControl-
means as EvoControl uses ES to update the low-level policy, the high-level policy now receives less equivalent updates compared to the high-level policy of the ovo-trolootool baselines. We observe in Tables 14 to 18 that EvoControl even when limited to use the same number of low-level environment steps as all the baselines, on average achieves a higher normalized evaluation return than all the baselines fixed controllers and direct torque control.
Table 14: Ablation. Equivalent Number of Low-Level Environment Steps, with es_pop_____pppp
pop_siieee 16. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 6400 random seeds, with + indicating 95% confidence intervals. Returns are normalized to 0-100 scale, where 0 represents random policy, and 100 represents the highest reward achieved hy non-EvoConttol baseline in each environment. Scores bolded are greater than 100.
Sume PPO Segh alc. with
Ani
Haltcheeuhs
Hagper
Humaanid
Hamancid Stundup
laverted Double
Inveried Pandlum R
Pusher
Reacher
Reucher
WalkerDD
Laww-Lrd Policy 5
Pendulam R
ID
R2
1
R
1
1
Fund Cont. PD Position
100::1.75
62.1500115
97.520.246
100:00.629
100800144
10010000754
1.0
100
100
$6:00.622
53. 33,140
Fiodd Cost. PD Position Delta
4.11:305503
2844100022
1001006699
967.:0.34 39]
2.75:00111
55.920.412
100_3377 07
33133.84
42 42.2200705
18.6&1.74
938810.022
Pond Cont. YD Velocity
2,05100304
2.5
76 16.130.319
0:10
48 00.35
36610.996
0.:0
01:0
01:0
Fined Cost. Random
0.0.0.00
0.010.00
0.0:0.0
0,0000
000:0,0
0,0:000
00:00,0
0.0000
0.01:0.
0,0:0000
0.0.0000
Direct longor Cont. High Proq, (50046))
0::0
24.18:223
12:3:0:356
01:0
413380434
0.34:0797
602:1.07
2:45:2254
17:68:122
72.66
96,4
Direct Tingur Cant. Law Focq. (31,25Hz)
46.5-142
100+0.350
783340:18
07.1-0
$1.3+0.002
100+-0008333
907H000276
80.6+9386
50.8-0.301
100-0 5044
100H0,202
ExnContee (Full State)
145 145-2.51
69.7+0661
127-0.996
9522-0651
889-00111
100-0.003
100-3.744-07
423--3.41
107-0.192
186
134-0.491
EvnCoutred (Reiddall State)
124:2.45
1255
83.580.719
117:0,768
101:0.159
99.6:0,119
100:33744007
445::361
105:0.178
105:0.227
188:0.311
ExnConteed (Targrt Proprin.)
148 148-1.81
99:2+0.71
146+0.998
122-0.707
99.1+00452
99.7-0.0048
95.4-0.70
468-3.31
107-0.177
106-0018
132:00.98
EvnCoutee (Target)
141:2.79
106::1993
135::0621
124:0.713
100:003374
99.9:0,3148
100:33744-07
4.11
65:3:0.302
103::.29
137::0.97
EvoControl (Learued Gains)
79.88:243
52.31.0457
109-111
85.3-0.752
93.6:10.178
72.5=124
100 07
409.3..5
99.320.421
97.9.00449
114::0.999
ExoContro (Delt Position)
139-2.81
99,210885
141:0,382
97.620764
683800.6
07.6000555
100 07
396:3332
632200802
102:00337
150:9.5399
Table 15: Ablation. Equivalent Number of Low-Level Environment Steps, with ss_poo_size = 32. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 6400 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 0-100 scale, where 0 represents a random policy, and 100 represents the highest reward achieved by a non--vooo nonnEvoConttol baseline in each environment. Scores bolded are greater than 100.
Same PPO high-lesel alg a with Lave Policy ou
Ani
Halfchrzhhh
Hopper
Humomnd
Hunommid Suadlap
Inverted Double
Invetall Pendulum Rt
Proder
Bescher
Reacher ID
Walker2n
Pendulum 1
1
R.
R2
1
R-1
R-
R+
Find Cost. PD Pustion
100:11.23
39.6100113 195
09.5-000311
83:2-00473
100:00:275
99.910,00044
100 07
100622.1
100/10.255
8633:06645
74.8:10125
Fired Cont. PD Postion Delta
3,4240.447
2724000204
100:0
961-0941
3030000777
100133744
50.73373
42 42.00000002
1..5
933:40.022
Food Cont .D Velocity
482:10.433
2.45-00214
81.5+0 702
09.4-0225
0+:
506-00404
969-00235
0:10
0:0
010
100+0.519
Fiand Cont. - Random
00:00.
00:0000
000000
0.0000
0.:000
0.5000
00:00
00:000
0.0000
00:0.00
0.0:0.0
Direct Toque Cost Fony (500852)
0:00
44.31:229
33.5::0499
55:00664
34.71:047
92.7:0.267
3.92:2235
77::113
47.5::0.004
Divect Biquue Cost Low Freq, (31 25882)
42 12.5:1.15
100::0257
99:410-46
100:00449
96.3:0.133
1061000002
10043.774 07
91.8 11.8:33.56 3.55
62.6 1.3.79
1001
$6.120
EvoControl (Fall State)
1.4
143:-05502
1435
93:00.63
96.5.00144
100100000077
300
411
107.10.163
106:1.117
149100084
ExoC (Resduall State)
1811
112:0.433
94.2100114
112:0.68
123:0483
99.530.0196
10043324e-07
42413.31
97 9774:00.35
105:00.29
20010.515
EonContral (Target Preppino)
152+2.39
120--1548
127-1L427
123-0.614
0.405
99.3-000001
100 74c-07
444+3.27
57. 57.3+0.813
106-5
145 44.79.
EsuControl (Target)
154:2233
103:-0.511
141:0.39
122:0.579
117:00409
88:420.494
1001:3.74--77
4044:3.29
96.410.309
104:01219
136::0.557
EvoContrel (Learned Galas)
104::2.1
67.100518
155:
96.920.557
100::0.209
96.220.42
1001:3.44 07
360::29
102.10.381
102:0.31
161:00705
ExoControl Poction)
161:2.53
105:00.07
127::0.1
101:00474
74:00479
100:0.0100
1001:.74e-07
429.:3.34
59.530.744
102:0.316
109:0.66
37

---

EvoControl: Multi-Frequette Bi-Level Control for High-Freqqenny Continuous Control
Table 16: Ablation. Equivalent Number of Low-Level Environment Steps, with es_ pop size = 64. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct tonque control. Results are averaged over 6400 random seeds, with + indicating 95% confidence intervals. Returns are normalized to 1-100 scale, where 0 represents random policy, and 100 represents the highest reward achieved by a non- non-EvoConttol baseline in each environment, Scores bolded are greater than 100.
Stnne PPO bightlesel als p LLow-Level Policy 3 of
Ast
Hallchastah
Hujpar
Hamanoi
Hannnoil Standop
Inverted Double
Inverted Pendelum
Pudias
Rescher
Reacher
Wilker
Pendalum
ID
f
R.1
R2
1
Fixed Cost. PD Posttion
10015.17
06.50.555
00.21...899
92.8.12.9
87.8.113
99.600
1.0e 06
100.9944
10011:24
$8.7....
68.7.00.255
Fized Cont. PD Position Delu
4,92&167
2.88000091
84.3400854
10021179
3.164:0024
57.141.6
1.55:.533-40 06
344999
42.842.99
27.927.48
85.420.513
Fixed Cunt. -DD lnt. Velacity
5.94+188 8
2.50=0.0935
56.9+2.
72.6+00822
0#40
61.3+11.99
99:11445
0=00
0+:0
9.58-7.72
100+:06683
Fioed Cost -anddm
00+00
0.+00
0010000
0.000.0
00:00
00+00
0.+000
0.000.0
0.0+000
0.0-0.0
0.0++0.0
Diovet Tonqur Cunt. -Hih Feeg. (500H))
0+:0
44:00,0 704
42.9:0.55
63.6+2.37
53+2.15
97.5+01387
49.0+2.00 36
9.18+7.60
10.5::5:42
51.1:4.24
32.2+1.27
Diwe Tinque Cont. -ow Freq 25Hb)
52
100+ 1:22
100+ 26
87.2+1135
109+0 704
100-0.0176
100+1.79
EvoControl (Pull State)
188.112
165::115
118::22.4
111::15.9
90.2:66.6
100
100:10
2961:42.6
109
104::2.14
127:1126
EvoControd (Residual State]
152:117
166::66.9
125+118
10M:43
112::14.2
99.2+00304
100::0
302:118.1
102:1100
104:2247
169::99.6
EvoControl (Target + Proprin.)
183-69.2
135++11.
130+55.2
126-12.11
124+558.8
87.1+53.3
100+0
300-42.6
1077-2.7
104-2.23
130+-12.7
EveControl (Target)
164
99.511.7
286.27
83.5163.4
103
1221.38
EveControl (Izarne Gains)
103+23.1
90+2
141+846
111-28.9
102+1.5
93.9+26
100+0
263--6.88
104+11.4
102-4485
149+38.2
EveControl (Delta Position)
16812999
123175.2
95.91400.8
94.218.68
100.00491
300:661.6
59.55.90
99.9.7.07
160:1124
Table 17: Ablation. Equivalent Number of Low-Level Environment Steps, with es_pop_size = 128. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 0-100 scale, where 0 represents a random policy, and 100 represents the highest reward achieved by a non- -vo--voCootool baseline in each environment. Scores bolded are greater than 100.
Same PPO high- level alz with Low--l Policy
Ant
Halffcheetah
Hopper
Humanood
llumanned Stantup
Inverted Double
Inverted Pendulum
Pucher
Rtocher
Reacher
Walker
Pondulum
R
R1
R
R
R 1
R
1
R
R 1
R.
R
Fixed
Position
10019,19
732210.419
861.0.57
89.5:12.72
78.4:21.31
99..10.0353
100:1.53e-05
1001.667
1001:0925
86:22.76
1001:0935
Fixed Cust PD Prodtion Delts
3.35-2.50
3.12+0. 101
48.8+1.0 1.06
89+ 69
3:22+00153
70.2-1 1.05
100+1.550-66
10:24-66
43.6+2774
19.9+7:52
72.7+1.51
Fixed -PD Int. Velocity
8.1222.0
2.82:10.10
12221:96
97.810999
0:10
50.2:1
01:00
0.881
85.310.32
Fixed Cust. -Raddom
0.0-0.0
0.0-0.0
00000
0.0+0.0
0.0+0.00
0.0-0.0
0.0+00
0.00000
000000
0.0+0000
0.0+0.00
Direct Tongue Cost. THigh Freq, (500
1.22154
66.5.2222
66.61:44
75. 75.7::2.44
93,4:00508
49.6::2:28
35.51:521
83.814.27
41.1:11755
Direct Zongue Cosr. Low Freq, (31,25112)
53.9 73.977.7
100::1.44
10013.31
100
1001:0.397
100:000029
100.1..5e 06
85 2.6.6
68. 08.112.8 89
1.55
98.51:0.244
EveConttel State)
-8.2
148-68.4
105+2.16
87.1-8
97.1+1334
101-93
100+0
252+ 16.8
106-0066
105-2.23
127--57.7
ExeContral (Residual State)
497:
191:33
94. 94
120::00.9
421::42
99.8:00.474
100:10
239: 18.6
102::4.38
164::6.2
128::127
EveContrel (Targgt Proprin.)
274-75.2
202-144.7
121+51.5
127:19.1
152+89.5
90.6-37.6
100+0
253+-44.6
107+1.27
105+2.33
135+-23.9
EveContral (Taarge)
313:6,88
204:2.9
113::47.8
122:: 10.3
135:99
90,4=31.5
100:00
268:
101::2117
164:1127
1281553.9
EveControl (Learned Guins)
116.1.44
142.194.7
104:18.6
10412.93
99.6_3.18
100:00
236.121.6
10519962
100-5.11
1341555.8
EveContral (Delta Pusition)
296: 14.1
151:50
61.51287
98.8::4.44
90:3:1245
1.1-0.13
100:10
238:
60.21124.1
95.6:119.8
123::41.4
Table 18: Ablation. Equivalent Number of Low Level Environment Steps, with es ptp_popppeeeeppeepppppppp = 256. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to (-100 scale, where 0 represents a random policy, and 100 represents the highest reward achieved by a EvoConrrll baseline in each environment. Scores bolded are greater than 100.
Some PPO high-lovel alg o Low Level Policy of
Halichoethh
Hopper
Humanodd
Humanoid Standop x2
Invorsed Pendulum
Pusher
Reacher
Reacher ID
WalkerDD
Pendulum
x²
1
R
1
12
X
f
R
Fixed Cant -PD Position
100:5.60
601:0-402
47.3
100+
96.2 628
100:000359
100:1 1.5--65
100 96
100:00.99
$5
100
Fized
Position
3.11866
2.71100777
27.6110.41
98.311.56
3.0300024
96.310884
100L1 06
847.18211
49.412.412
21.917..1
57.2.2.07
Fixed Cost. PD ist. Velocity
6.16:1..9
2.461:00866
24.5:0993
57.3
0.:0
22:0.756
100::1536 06
0.00
0::0
3.33
66.1201156
Fiaed Cuni Random
0.0:0.0
0.0000
0:0:00
0.00000
00000
00:000
0.00000
00::000
800+000
0.000.00
0.0:000
Direct Torque Cont. High Preq, (500Hz)
26.7:13.2
6..261.3
58.720.756
65.722.22
75.14223
99 99.5:000562
60993.02
19.3:55.42
45.71:4995
$3.41:3.99
43.421.66
Divvi Tongue Cont. -Low Frag. 25Hz)
52.455.84
100+00.99
100+0.664
94
100
100-0000422
100+1
89.8939.
81.242241
100+:1.54
79.1-00946
EvoContred (Full State)
302:440.8
144:
09:9:182
17.1
109:19.4
101:0.524
100e0
206:115.8
89::64.6
103::2.1
123-:379
EveContral (Residual State)
138-28.7
162+22.8
37+3.31
124-49
129+:37.4
101-0.561
100+0
215-21.9
90.9+7.45
103+
128-94.11
EvoContre (Target Proprio.)
281:3905
130::7.09
90.1=11.2
125::42.8
141::70.1
99225500
7699999.
205:111.
104::3.9
103:2214
118:3007
EveControl (Target)
241-42.1
150-19.8
75.1+1982
124+
118+8.06
98.7+2188
100+0
198-21.5
75.7+56.7
102--200
92.7+30.3
EveContri (Learned Gaims)
197 16.2
96.311399
99.115099
111:21.7
10711.18
101.00.02
10010
1901145.8
104.00.93
10212.65
135.440
EvoContro (Detta Positioa)
357
116.123.7
102.:2.25
98.41:14.1
98.5111.2
101-1.55
10010
206-125.4
65.31:40.5
95115
130.:48.7
38

---

EvoControl: Multi-Frequeen Bi-Level Control for High-Freqqency Continuous Control
J.4.2. EQUIVALENT NUMBER OF SEQUENTIAL LOW-LEVEL ENVIRONMENT STEPS
Here we set the total number of sequential low-level environment steps for all the baselines to be the same. This is approach is different from setting the total number of low-level environment steps to be the same, as it acknowledges the more realistic scenario of performing parallel environment rollouts modern GPUs. With parallelized rollouts, the computational bottleneck becomes the number of sequential steps within each environment, as the overhead of increasing the number of parallel environments is negligible compared to increasing the number of sequential steps-assuming a user has a sufficiently large GPU to perform parallelized rollouts of the environment. Such an assumption is common in practice (Huang et al., 2022), with many implementations of PPO and simulation environments natively supporting parallelized rollouts for the environment (Brockman et al.. 2016b).
To set the total number of sequential low-level environment steps for all the baselines be the same, we follow a similar setup described in Appendix J.4.1, now only accounting for the sequential low-level environment steps that EvoControl uses. Specifically, we first compute the total number of sequential low-level steps that EvoControl uses, where we train EvoControl's high-level policy for IM steps, and then now train the baseline methods for this increased number of equivalent high-level steps. Specifically we consider a PD position baseline, originally training this for 1M high-level environment steps, with a lower-level PD position controller, operating at higher frequency with = 16, meaning that the number of low-level environment steps used the environment are 1M x 16 16M.. Here EvoControl, when the high-level is trained for 1M steps, the lower-level policy also receives updates, therefore the total number of sequential low-level environment steps used within the training of EvoControl is K x es_ sab__generation x episode_lengti As es es_rollouts and es es_pop_size are parallelized, they are not counted in the total number of sequential low-level steps, therefore we leave them as the default values as defined in Appendix G. We leave the other parameters the same as they were for the main results (that of K = 8, es_ es_sub _uu_geenratiios = 8, episode _engthh-eehhhhh-e = 1000). This leaves the input parameter of es ts__oo__size that we can vary. Therefore the total number of sequential low-level environment steps used by EvoControl is 1M x 16 + 64K, fixed amount. Therefore, for the following experiments we train all the other non-EvvConttol baselines for 1M x 16 + 64K level environment steps, by specifically determining how many high-level steps this is by dividing by G and using that as the input the total number of high-level steps to train for each baseline.
For the results, as discussed, we vary es_ es_pop__s___eeeepeepppppppepppee size = {16, 32, 64, 128, 256} and re-run each non-EvvControl baseline with the equivalent number of sequential low-level steps as EvoControl- means as EvoControl uses ES to update the low-level policy, the high-level policy now receives less equivalent updates compared to the high-level policy of the non- non-EvoControl baselines. We observe in Tables 19 to 23 that EvoControl even when limited to use the same number of sequential low-level environment steps as all the baselines, on average achieves a higher normalized evaluation return than all the baselines fixed controllers and direct torque control-which remains consistent with the results seen throughout.
Table 19: Ablation. Equivalent Number of Sequential LoweLevel Environment Steps, with es_ es_pop_siee size = 16. Normalized evaluation return RR for the benchmark methods, across cach environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds. with + indicating 95% confidence intervals. Returns normalized to 0- 100 scale, where represents random policy, and 100 represents the highest reward achieved by a non-En non-EvoControl baseline in each environment. Scores bolded are greater than 100.
Some high-Sevel p Lowo Level Policy of
Halfchoetah
Hopper
Homanoid
Homanoid Standup R
Inverted Doable
Inverted Pendilum R 1
Pusher
Rencher
Reacher ID
WalkerDD
Pendulum x²
X²1
1
R
R 1
R
X 1
R
Fixed Cont -D Position
100-77.29
69.3
60.3+0.723 723
93/7=1.26 1:28
100+2.75
100:0.194
99 90.9:0.0347
100:1 100:1153e-06
1530-06
100=8.47
100+:1.9
85+2293
62.8-00. 176
Fized
Position Delta
4.81.1.78
101
1.05
97.7.185
53.112.1
1.0L1.53
40.311119
15.2__7.6
93.3.00.335
Filed Cout. PD Jnt. Velocity
4.8::182
28:0.106
75.2200855
$41:134
0:20
48.8::193
17.8
01:0l
0::0
0:10
34.1=3.36
Fixzd Randons
0.::0000
00000.0
00:00.0
000:000
0::0000
0.0=0.0
0.0:0000
0.800
0.0:0.00
0.0=0.0
Diwer Tongue Cost. High Freq. (500H))
DH0
195340.355
1.85-0273
134422.2
15.3-1.23
0+0
D+0
2,05+7.81
0+0
45.3+6774
1.15-4.13
Direct Tongue Cont. -Low Freq (31,2
55.6116.99
100.2..73
75.910.466
89.712.27
71.9.2.3
100_0.041
100L1 06
75.41 8
59.213.7
1.0.
100.10.384
EveControl (Full State)
166.622.6
104:22
127:52.3
94.71132.6
6.4
0,55
100-00
272:1112
95.5165.5
105::3.1
130-6639
EveConteul (Residual State)
163:118
153.55
96.515.44
93.81140.1
95.6.130.9
99.212.11
100_00
314
107.1146
105
194.33.2
Evol'onteol (Target Propeis.)
121:549
92.1=20.7
131:1669
117::10.2
101:3.66
994:12.5
1000=00
311:5998
110:5576
106:2.39
137:1104
EvoControl (Target)
129-778.
115+
133-2.96
110+11.11
87.8-5.32
100-2.15
100-0
303-5332
71.3+46.9
164-4991
143-382
EvoControl (Learee Gains)
468:154.2
58 68.628222
128:748
82.617706
91.1:8877
7566::19
100:00
225:41.4
103:1339
97.3:11.76
EvoContrul (Delu Position)
117.13388
104_557.3
154.41.2
81.9123.5
67.6.566.7
98.1124.1
100_0
285152.1
58.6.6677
99.713.16
135_.68.1
39

---

EvoControl: Multi-Frequenge Bi-Level Control for High-Freqqency Continuous Control
Table 20: Ablation. Equivalent Number of Sequential Low- Environment Steps, with es _soppppp___. size = 32. Normalized evaluation return R for the benchmark methods, across cach environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with = indicating 95% confidence intervals. Returns normalized to a O- 100 scale, where 0 represents a random policy, and 100 represents the highest reward achieved by EvoC-rooonool baseline in each environment. Scores bolded are greater than 100.
Some PPO high level alz. with
Ant
Haiichertah
Hopper
Humazoid
Humanoid Standap R1
Inverted Double
Inverted Pendulum
Pusher
Reacher
Reacher ID
Walker2D
Loow-Level Policy of
Pendilum
RR
R1
R1
R1
Rt-
R1
R1
R1
Fixed PD Position
100.15.89
0,55
895.112
39.213227
100L133
99,81000061
10011:53e 06
100.8.47
100.1188
8512.95
76.810 193
Fized Cost. PD Posltion Delta
2.14-1.8
2.77000991
1001199
100612
2.73:000726
53.92211
1002 1.551 06
41:13.24
15.2:7.6
896640.17
Fixed Cont. PD Int, Velocity
182.1156
247.100037
75.0.882
1,21
0.10
48811193
86.412.01
0.00
0.10
0.10
81.313.18
Fized Cost. Random
0.0:000
0.0:0.0
0.00000
000000
0.00000
0.1:0.0
00200
0.0:0000
000:0000
0.0:1000
Direct Torgue Cost. Iligh Ureq, (50010))
0.10
16.900,325
3.5410225
2.2812.31
21.911.00
0.10
0.10
2.9517.81
0.10
45.3.16.4
0.10
Direct Toruue Low Freq. 25tt0
45.124.76
1006117
79.36 22
93.7422.47
69.6:22:38
100600041
1002
103-1-53
73.12:2.7
59.1:3377
10041194
1006:0637
EvoControl (Full State)
150.146.9
147::10
1321:27.3
97.8
105::17.5
100:00451
100
305.33..8
115:22.66
105:11.89
138.1:56
EveControl (Residual State)
155
11.3
93:714442
116.119
12111993
10011.45
100.00
333160.3
SAL18669
1081:M9
161.1327
EvoControl (Targel + Proprio.)
156:133.6
111::49.3
1411:20.5
126::12.3
117
94.3200367
100
324::40.8
55.122.89
106::2.35
137.1799.6
EveControl (Targed)
164.5565
116 2,59
136.127.6
129.44.4
110116.7
96.2112.2
100..0
291-104
88.7.162.8
105.0.566
139.23
EvoControl (Learned Gains)
86.3134.
59.9:1118
1621:71
105::18.5
103::21.7
98.11112.1
100
303:557.5
110::6.3
102::4.43
155:1100
EveCoutrol (Detaa Position)
156=78.3
101-35.5
111::49.3
101-7.07
99.74803
100:3012
100=00
58.945.31
102::9445
127:560.5
Table 21: Ablation. Equivalent Number of Sequential Low-Level Environment Steps, with es psppppppp_.. size 64. Normalized evaluation return R. for the benchmark methods, across each environment. EvoControl on average achieves higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 0- 100 scale, where represents a random policy, and 100 represents the highest reward achieved by non-E -on-CvoContool baseline in cach environment. Scores bolded are greater than 100.
Same PPO high level alp p with
Azt
Hallfcheetah
Hopper
Humanoid
Iumanoid Stundap 12.1
Inverted Double
Invected Pendalum 32.1
Puster
Reacher
Reacher ID
WalkerDD
Low Level Policy of
Pendulum
12.1
12.1 1
X211
R2.1
12.1
32. 1
R2 1
22
Floed Cont. PD Podition
5.0
64,21.0518
91.8.1113
981277
10010.123
99.11005
100 06
10018.47
100118
$5.12.93
82:2100371
Fixed Cosg. PD Position Delu
3.66
2.92000928
10011155
10011.79
2.4910,0711
53881137
1.03 06
0:10
41::3.24
15,2-7.6
99.7100265
Fioed Cust PD Ist. Velucity
4=1.56
2.57:000076
75:24:0955
90.11148
04:00
49.7&1.55
86.5222
0:50
04:0
U0:0
93.8:33003
Fixed Coog. Random
00000000
0.0.0:00
0.0.0.00
0.0:0.00
000000000
00010000
0.0000000
000:00000
0.0.1:..0
0.0.0.0
0.0.100.0
Divee Toquee Cust High Fecg. (500Hz)
04:0
18.240.332
13316:0536
39.3:422
11.499992
04:0
0::0
4.36::7.75
04:01
45.3
Diover Tonger Cunt. -1ow Freg (3) 25Hlv)
48 5.23+5.32
100+1 1.64
77.5+-0571
05.6+2.66
74.5+1.76
100-0.0111
100-1.5h-06
75.4+12.8
50.14-375 3.75
100+1.94
100+-0.999
EvoControl (Pull State)
175:55.9
163116.17
1481132.6
115::25.4
103:127.3
10110.306
100.00
3481
1141:595
106:12.17
166.185.1
EvoCCouteo (Residual State]
143::33.5
163::32.4
170::362
144::7.1
116::10.2
99
100::0
349::8.24
105::1.2
105:3387
22M::27.1
EveCCantee (Target Proprin.)
183+124
126+:57.2
164+488.3
137-11
114+:1277
87.345344
336-4482
114+2293
1.6-2233
157-+51.3
EvoCoutted (Targel)
180:
176::12.8
132::1.4
138::619
126:166
99.7411.7
100:30
326:141.6
91.24472.9
105:3325
157::79
EveCoatteo (Learned Gains)
107+26.7
91.2+2248
196-115
110-29.7
104+
94.1422611
291-19
112+6.83
104-1.37
211-37.8
EveContral (Delta Position)
180:18.7
117::25.1
1311:36.6
112:9984
59.41.30.5
101:00492
100:10
342:70
61.319332
101:7.19
189::36.7
Table 22: Ablation. Equivalent Number of Sequential Low-Level Environment Steps, with es es_poppsiiee sizeiizee = 128. Normalized evaluation
return 'R. for the benchmark methods, across each environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds. with + indicating 95% confidence intervals. Returns normalized to a 0- 100 scale, where represents a random policy, and 100 represents the highest reward achieved by a non- con-EvvConttol baseline in each environment. Scores bolded are greater than 100.
Sume PPO high-tevel
Azt
Hallcheztah
Hopper
Humancid
Hummoid Standop t
Invested Double
Invested Paudulum
Pulter
Reacher
Reacher ID
WilkerZD
LLon-Leve Policy of
Pendulum
R
1
R
13 1
R
Cont Cont- -PD Position
5.0
6..5:00452
82:2:1.12
100-2.82
100:0.151
90:9:0.03
10001 .53--06
100+:8447
100:1.8
25
75.5=00.0 396
Fized
Position
3.7511.68
301_0.0975
1.17
24110,044
53.811.57
10011
41.113.24
1522176
92.9.0.185
Fixed Cont. -PD In. Velocity
181116
2.7:0.103
67.2-0.812
81.4+1.20
01:0
49.7:1155
$6.5+22
0::0
0:10
01:
86.5-2.67
Fized Random
001100
00100
0.0.0.0
0.0.0.0
001000
0.010.0
0.0.1000
00100000
0.0.10.0
0000000
0,0_00,0
Direc Torgue Cont. High Free, (500Hz)
01:0
13.5
1.77.0.79
1011100
15.910.799
010
2.9517.11
0:00
45.32.6.4
0=:0)
Dewd Tongue Cont. -Low Reg. (31,25Hz)
3686492
100=2.11
63.2=0.505
90.4=2.1
87.4+100 02
100000311
1005 1.536-06
73.1=1177
60.6
100+100
100+004166
EvoControl (Full State)
258:660.9
124:1131
157:62.4
9944=13.6
93.445777
101:001993
160:=0
346::3.75
115:3334
106::2.28
184::4655
EveContrul (Residual State)
166:18611
180-239
125:1163
147.177.9
1401:1.5
99.8104474
100:00
300::30.4
109:4407
104:5595
211::19.8
EvoControl TTarget Proprio.)
241::45.1
198:44
172=6899
139:13
134::06
90.6:37.6
100:=0
363:112
116:2.5
106:
163-69L3
EveControl (Targef)
218+-5.11
107-5,49
165-93.1
132-2007
134-99.5
92.4+31.5
100-0
335-440.4
109-2.03
105+1.22
155-21.1
EveContmi (Learned Gaine)
165.15.1
107 10.1
170
124.
98.1117.4
99.713.18
100_00
3997
112::8.6
10415.15
67.8
EvoContel (Detaa Position)
216:130.1
125
114:1133
115::12.1
94.112999
101::0.13
100_00
3341:16.5
59.4116.8
95.1121.7
190_2747
40

---

EvoControl: Multi- Multi-Frequeccy Bi-Level Control for High-Freqqency Continuous Control
Table 23; Ablation. Equivalent Number of Sequential Low- Level Environment Steps, with es_pop_size = 256. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with = indicating 95% confidence intervals. Returns are normalized to a 0-100 scale, where represents a random policy, and 100 represents the highest reward achieved by a non-EvoControl haseline in each environment. Scores bolded are greater than 100.
Stme PPO high-level alp. p with Low-Lovel Policy of
Hallchaztah
Hepper
Humonnid
Humanid Standup R-1
Inverted Doutle
Inverted Pendilum R1
Pusher
Reacher
Rezcher
Walker
Pendulum
ID
R.
R
1
f
T
72-
R2.1
R.
R-
Finnd Cast. PD Poxition
100++6.25
63. (3.2-00599
89.3+12 1.2
92.6+2005
100+-0200
99, 99.9-0.03
100 180=11533448 1,53-466
100+-8:47
100+:1.77
85+2203
$1,7+0 384
Fiadd Coot. -DD Posiiin Delia
2.57
2.86:00.002
100:1.54
100
2.54
53.8&1.57
01:01
40.8::3.23
15.247.6
96.34:0219
Finnd Gast. PD les Vilecity
2.46+1.51
2.55+0/0667
73.140844
$6.6+1135
0++0
40.7=1 1.55
86.5+2
0+:0
0+00
0+00
98.722.77
Fiued Coot. Randon
00:000
0.0-110
0.0:0.00
0000000
0.0::000
0,000000
00:00.0
00:000
00000
0.00000
0.0:000
Diavt Tonpur Cost. High Fraq. (5008t))
0:+0
18+0.353
0.882-0,535
7.93+1994
10.1++0.357
0+00
0+0
4.75
0+00
45.346674 74
0++0
Dirus Trquur -Low Flng (31.25984
634707
100:=1.54
70.6:0.544
85.66:141
90::1.73
100+0/0311
100t=1.53c-00
75.4
58.843374
1002:194
1006
EveCuntrol (Full State)
288 255,7
161:19.5
190:440.5
126.99.2
105:
101:0.522
100:00
14.4
97.91170.9
1061:2.6
1691158.1
EveControl (Residual State)
1611149.5
95.7155.
153.113.7
17511160
100.0.555
100.00
3513:54.3
112::1.46
1051.299
20416003
ExeCantrol (Target + Progerio.
293:108
134
202:113
168: 16.2
1401766.2
98.8::5
76 76.9199944
3:8:145
114::376
1061:2.88
1791130.8
EveControl (Target)
3485
175
1521.3188
150.1145
99.31.2.17
100.00
146::146
95.612238
105.2287
145131.3
EveControl (Learnd Gaim)
214:112.7
111:11.2
128
125:72.2
102.18.52
100.00.201
100:00
827:
115400874
105.1267
2291556.9
EveControl (Detu Podition)
251,176.8
141-31.4
202:888
121:88.77
96.3116.3
101::1.56
100.00
342:12.7
71.9144.6
97.4115.4
1731332.4
J.S. Ablation: No annealing with PD Controller
Here we conduct an ablation of removing the gradual annealing with a PD controller throughout training. Specifically, to do this we set a 0, therefore meaning that within EvoControl the high-level policy P output latent action a directly goes into the initially un-trained low-level neural network policy (3. We observe in Table 24, that EvoControl as presented in the main paper, the inclusion of annealing with PD controller throughout learning does help EvoControl to achieve a higher normalized return on average compared to no annealing with a PD controller, as shown in the ablation. This provides empirical evidence for it's inclusion, which could be explained by the intuitive arguments presented in the paper, of helping the higher-level policy p to learn a stable policy using an initial goal- -rackkakkkin sub-policy of PD controller, and then switch to an improved learned low-level controller throughout training. We also note, that although EvoControl performs well with the inclusion of the annealed PD controller, not having the PD controller, it also performs acceptably compared the baselines. However, for best performance we recommend users to use the annealing with PD controller.
Table 24: Main table of results (Table 3), with the ablation of EvoControl with no PD controller annealing (by setting a = 0). Normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 0-100 scale, where 0 represents random policy, and 100 represents the highest reward achieved by EvoConroolll baseline each environment. Scores bolded greater than 100.
Sors PIO high-teed dp path Low-Leval Poicc of
Ant
Hepper
Harowoid
Berrassid Stanlap
TremmeDarhle Darbile Peslalan
Inurrel
Pricker
Beocher
Rowter ID
WilbertD
Pealalas
70
R1
R.1
72
Fowt Cart IDD Poitton
19116.56
916.1122
1.0
100:1553e
1000
100.18
25222287
15.7-0055
Finnd Poriion Deto
244119
2 0688
100:1.35
76
296:00397
53.96157
100:1.53e-66
060
152-16
90.2-0.299
Wasee
Findd Cosl. Raskon
00:00
00:000
00:00.
00:00
00400
00500
005000
00:00
0.0e80
00:00
0.0000
Dive! Trager Car. High Fieg (5006))
0:00
17240336
14246.533
10442.19
103+0.50
010
0:0
1.346730 50
286-55.4
45
3536674
020
Diver Tinger Cosr- Low Fres, (3122M15
54527715
100:3.21
7240164
5542355
91642.56
100-010111
100:1333340 06
33.20129
30.2 3.72
10001 16
1002240 66
ExoCasted Sutoo
304-733
137:1833
374:2066
133:19
116118
101:0-55
100:0
362:12.5
114
106-275
103-1137
EscConted (Rediull State
182-16.1
183:4.31
101:5.51
170- 15.2
212-1485
59 59.74125
103-0
375 91.1
255
181-3342
215 47
ExcCasted (Taract Propein.)
309:35.1 1
168:
171:115
165:7.19
166:150
99.71119
100:0
353:2884
96.5:78.5
1055471
178563.7
EnoContee (Targel)
243:97
266:147
353:4344
112:1173
105:146
188-98
EveContrel (Dota Pailisni
M014477
225:523
119:18
10534.64
101:00354
100:0
267
99.12127
100:344
Alaiinn cerodl stceeling (ir ExaContrel (Pull
349-15
ANarin Po zereatin (o (I) PasCantral (Redaul State)
164 195
131 21.2
168-181
166-28.2
12212566
46711447
100-0
364 86.2
10217
185 10
Alaaion No PD cotade usssing (ir (). EvaCasteel (Tergel Proprin))
29::46
145:3302
242-193
172:184
166:147
39.354333
100:0
357:44.2
13.2::12.
1041:334
198:1106
Alaio No P0 oeroile snceiling (i 0) EvaContre (Tergel)
201-315
145:7114
121:100
(74:5.8
129:1666
68.76781
102:-0
156:109
74.4:41.3
91.61.99
1951:744
Akaiin No FD cerndle stzealng (i EveCantrel (Leumal Gairo)
258:58.
$48:115:3
952=12)
156:13.9
11913.95
100:0.554
100200
348::4944
67.922401
92.724455
179:1117
ANamin No creniie strealing
Costral (Tieta Poditon
336 110
151 151.
255 116
127 1
121/873
IL165
107. 0
36.9
02-11.3
96. 96.11153
103
41

---

EvoControl: Multi-Frequency Bi-Level Control for High-Freqqency Continuous Control
J.6. Ablation: Main Results for More High-level Steps
In the following, we increase the high-level number of steps that each baseline is trained for. Initially in the main results presented in the paper we trained all the results for IM high-level p policy steps. Therefore we ask the question, how do all the results compare if we run all the baselines for the high-level steps of 100M and IB high-level steps. We tabulate these results in Tables 25 to 27.
Table 25: Additional Experiment. Training all the baselines for larger amount of high-level p policy steps of 10M steps. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized 0-100 scale, where 0 represents random policy, and 100 represents the highest reward achieved by EvoC--ntoll baseline each environment. Scores bolded are greater than 100.
Sune PPO bigl-
Halfcheetah
Hupper
Humanodd
Humanoid Standup R1
Inverted Double
Invested Pendulum R.1
Pusher
Reacher
Reacher
WulkerDD
L. Level Policy 3
Perdulum
ID
R
1
R1
R1
Find Cost. PD Position
60:81:201
08.0:0.555
10061.23
100
1006:0.308
100::00033
89,6:1144
100::499
93.8=2.48
84.9:2.56
1006:0775
Fioed Cost. PD Postion Deka
D1:0
4.13:00100
45 45.4:00054
86.5::122
1,49
98, 08:22::455
100 06
0.236
05.5:1..6
25.7677.17
46.55000129
Foed Comt.
Velecity
4,15:0.54
1344:0.33
Frand Coni. - Random
0.0+00
0.0-00
00:0.0
0.0000
00:00.0
0.0+0.00
0.0+0.0
0.000.0
0.:00.0
0.00000
00:00.
Divect Toquar Cost. Freq. (500H))
0.36:4.11
70:1.51
35::166
42.181.2
73:4:1191
97.8 148
42.621.48
10.7::4.9
36.9339
83.8:33.45
2.:8:1118
Direct Teque Cost. Low Freq, (31 2512)
4.06:406
100:1139
94.61399
96.3-0.355
79:5:2201
100:0000366
100,11:53 06
41.723883
10021.33
100::1.66
74.8:2244
FxnCintral (Full State)
2854
131-1111
187--0629
104-0.802
118
102 -82458556-05
100++ S3c-06
223+3306
07.1+0
99.7+114
212-1.12
ExuCuatrol (Reiddual State)
113:3774
208:1
128::4.11
124-0.865
2.5
101-0,0453
100:1.533-06
228:33/13
103:8.646
101:0,634
190::0.901
ExoControl (Target Proprio.)
178
230:005542
177:00467
169-20.222
206::6.11
$3.6:1..6
1001:0000679
222::242
96.720.965
97,82:12
226:11.66
ExoControl (Target)
231-0617
$7.31.1899
06
334::284
927-=108
99:5:1004
200:11722
RxnCentral (1 sorned Gaine)
147 4.81
194- 1.13
172-11682
110-1.86
112:0.506
182 0.0892
100++1 SSc-06
200 200-4462
602-
90.8-0.836
185-1.58
ExnCrntrol Delia Posiliun)
1875-408
219:1.55
186:-1188
08.6=0.324
115:0.389
102:6.666-45
100+1.53.-06
165:5.18
03.4=1.03
08:4:0.777
179:00:35
Table 26: Additional Experiment. Training all the baselines for a larger amount of high-level p policy steps of 100M steps. Normalized evaluation return R for the benchmark methods, across each environment. EvoControl average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 0- 0-100 scale, where 0 represents random policy, and 100 represents the highest reward achieved by non-EvoControl baseline in each environment. Scores holded are greater than 100.
Sam: PPO higholoxel alg plg.poo with Low-Levol Policy &
Halfchaztsh
Hoyper
Humanid
Hamaoil Standup R
Inverted Deubie Pendulum
Incetal Pendulum
Puster
Reacher
Reacher
Walker2D
R2.1
R.1
R.1
2²
12.1
1
1
Fiad Cost. FD Position
62.5214
38.62:0485
$5.1:2..9
1001:0289
36:22.01
100.0.19
98, 18.1:004118
1001:455
100=1197
85.5225
75.7222.6
Ford PU
Delaa
2.34:0.31
53.910990
10910.0021
18.11188
20.4
46.330.0022
Fiand Cont - PD In. Velucity
261-1003
4 188
24942214
68.4+1.68
35+1.03
34.6-2.79
44.61-401 01
52.746--2
0-0
12:1++
52.4-1.8 1.14
Fund Cont- Rauhan
00:00
0,0600
0000.00
00:0000
0.00000
0.0:00
0.000
0.00000
0.0:00
00:00.0
0.0:00
Direct Toque Cost. High Feeq, (500H2)
29.8:4.11
1001:
77.1422.3
75.112.87
100:2.58
16.511.06
63.21:286
9.0616.19
$4.22.74
65.326.04
72.7:2265
Divect Tonque Coni. -Lo Frsg. 21,25Hz)
100:36
60.62:1.3
10045.62
96.3690501
1..7
99 09.930.281
100--153--06
72.56:4.9
92.74:3.2
100:2.19
100::3.95
EvoControl (Poll State)
1819
19150
137:10.36
159:1.64
19913.11
124
100
2781335
1022007708
102.19.577
264:11:06
ExnContro (Residual State)
3.86
151 2.44
186
16300
194-6.19
100 000-0
100+1.590066
28443..1
104-1.47
102 +0.86
211
EvaCuated (Target Proprim.)
155:3.15
192:0.655
216:11.9
192-1
241:4483
76..60226
100:0:1112
283:-2.66
163:0.782
96.8:1.3
257:1.38
EvaContcol (Target)
2.77
192:0.499
213:2.3
18911.66
261:22119
96.420.314
98 98.9:0.791
279:3325
96.921.03
85.1:2.05
260::0:733
ExoContral (Learned Gains)
117-346
1512:1.37
12310874
63.62226
100::42e-05
1001:..53e
259::4.17
163::08441
101:10:848
180_2..9
ExaContco (Delu Position)
180:3.45
186::100
325:11.32
1221:186
111::0.365
100::3.4:055
100::1.53e-00 06
253::579
104:0065
102::0.566
207:22.45
42

---

EyoControl: Multi-Frequeeii Bi-Level Control for High-Frequency Continuous Control
Table 27: Additional Experiment. Training all the baselines for larger amount of high-level policy steps of IB steps. Normalized evaluation return R for the benchmark methods, across cach environment. EvoControl average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 0-100 scale, where 0 represents random policy, and 100 represents the highest reward achieved by non-EvoControl baseline in each environment. Scores holded are greater than 100.
Same PPO hightleeel dlp pwith Low-Level Policy 3 of
Ant
Hallchoztah
Hepper
Hummnnii
Hunonoid Standep
Incerted Douhil Pendulum
Inverted Pendulum
Proderr
Reacher
Rezsher
Walker2D
ID
R.
R2
R.
R
R.1
R
R.
1
Flodd Cost. PD Posison
45,4:22.72
29.820.662
65.821.03
100200544
100_909959
98. 110.74
98.210,995
10011.6
100_11544
100
20.22:242
Food -9 Positon Delta
163:10459
50.911002
1,3,00,06
100:10.55
100
46515.00
26.6:1777
22.118.77
33.20000073
Floed Cont. PD Lst. Velocity
4,56:10.755
0.10
17.242.89
56.511992
36,50048
X2. 22.6622.23
19.6133.6
30.42204
0-00
6,5618.55
4.8121277
Fooe
Random
0.0:00
0.0:0.0
0.000.0
0.00000
0.0100
Direct Torque Cont.
High Freq, (50012)
40.823372
100111.74
56.312.36
$2.925399
13.8220.222
95.310.751
28.6=2.27
$2.916.85
Direct Torgue
Freq. (31 251k))
10013:25
74441:41
984110.557
EveContral (Fall State)
220::4.55
197::0.54
285:10.424
159:11.544
185:11002
101:5.
100:1 036
106:1.63
104:0.61
120:00.664
196::0.435
EveContral (Residual State)
13413.24
14912.34
270
1591
06590
101-0,000788
100%
95.3=241
120.00727
15524
EvoConteal (Targel + Proprind
182:33771
194:0.364
269:10.707
183:1.011
340:44,66
96.5++0390
100=1 36
114:1114
98.7:1.19
112::143
181:005355
EvaContrul (Turget)
1742:407
194
269.0.71
180
355:144
17,5
100g1.533
96.5-1221
112::1.5
198200428
EvoContral (Learned Gains)
87.3.1198
49.3
80.7.0.277
0.340
83.11288
101114.0681
999.10.204
76.9132.7
91.7-2.51
10312..4
08.91505
EveContral (Deia Position)
215:3381
195.0,499
26111.15
121::0536
158:1.55
101::434:05
100=11533
1124:1335
108.0.478
120:00.67
167:00.922
J.7. Main Table of Results Additional Metries
For the main table of results presented in the paper (Table 3), we also provide un-normalized results in Table 28 and the time taken to train the policies in Table 29. Regarding the training time, the wall-clock time for EvoControl can be substantially improved, as the PPO implementation we use to train the high-level policies is implemented in Jax, and was pre-compiled, across a batch of environments, using Jax based environments (Brax). Whereas the ES could further be compiled, however for our implementation it was not, and only the population of rollouts was compiled in Jax. Ideally for optimal performance the entire ES step could be compiled in Jax leading to speed improvements, however given that all the baselines, including EvoControl could finish training their policies within time interval of approximately one hour, further optimization was not necessary.
Table 28: Un-normalized evaluation return R for the benchmark methods, across each environment. EvoControl on average achieves a higher evaluation return R than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals.
Sunec PPO highlloe slp. LoveLard Pelicy 3
Halfinetta
Hoppe
Husuncid Standep
Innette Double
Invital Pezahiure
Pasies
Rescke
Reuchn 1D
WikkeD
Pentiioss
R1
R
R
R
x
R-
R1
1
R11
R11
Food Cont. Pretien
Latestts
-6852244
Findd Cas. PD Pootics Dela
Adcl A66
1144:082139
451e4042:55 5
6 6.18:403:103
1e:08:1
4.73600:2266
-12724.38
500255
Likes0 1o
Flott Coee 110 In. Vireety
Uilleoss 101
-2.1111744
leeffi 786
s VioeiTLi0 103
896.
-2807e601 26.2
-32017.55
447/547
Iitee33 2677
Find Crsy Banbor
0.:00.0
6n600
0.0:00
00600.0
0050.0
0.500
00:00
0.0600
D0:00
Direet Torge
High (500)))
920-4111
3734-580
119-55
559e-0d:-00 9W0
132-1 23
106-3.72
3.66661-444
-1943792
11
1250:002233
$482655
100e 103
9.220
00:00
244
-1045.04
L17e102:301
ExaContral (Pall
7l0w03 147
2 344
294 312
1lesst -n Tlesnd
5.55000- 56.5
1e-05-b0
56w61-3367 36.7
37.6-102
444:1.8 1.59
3 el Stee03
EvaCuairal (Besklud
185210827622
52061025455
9.10100211.2
1o:00ab
-151e102229
-345555
46602447
ExeCuatral TTagge Progric)
151
304
185e.03e1
115
283e 31c
0220036783
le:0000
3.58c100458
509
557433
20561026:166
EnstCnttral Barget)
431
3114011:300
342e4tt1:2 Iteett
-0182233
-37821.99
PaaContral Rearsed Galsci
106e-031119
476
1964033161
1e-05-10
463.77
254003-11 376403
ExeCustral (Dels Poitioa
2716103:226
243:1002544
40100
1.55 (25 103
927c:00:26
le Lo:00e0
-1 1.36:0022333
48.4:25.5
-10229116
43

---

EvoControl: Multi- Multi-Frequeite Frequency Bi-Level Control for High-Frequency Continuous Control
Table 29: Time Taken to train in minutes for cach baseline against each environment, for the main table of results in Table 3. All the basclines including EvoControl can train their policies average within hour. Results averaged 384 random seeds, with + indicating 95% confidence intervals. As the random baseline does perform any training, we put placeholder 0 for it.
Stnte PPO levh- with Level
Adt
Halfcheenai
Hupper
Hamazod
Hamazedd Stadtep R
lnveted Double Pendulari
Inveted
Puiler
Reatter
Reahhi
Walker2D
Peodulani
R2
R1
Prottans
0801:00012
0.374:0.20
0.5M6000173
0.30230011
0.597
Plaed Cosr -P Treirom Datis
Bo68 -0011
1006-6.0000
01575-10:25 705
1.110.335
0547-1000776
18775-0.00015
0.8710 N13
0.9011001
(1973 4001
16 CLO176
Findd Cosr. -D Vddnaity
0.802.000 00033
1.0600.0043
0.98100019
0,314.10.3
1:1110.00
0.31.00161
0.506.00.111
067910377
0,51LLDARR
0.50.0002
12:00:49
Dinee Targor Cont lligh Prog (100060)
8:97--00037
6:78100015
0:313:0:22
10010:217
0457-1000175
0426100131
0.79100281
0188-0062
10021
Oirect Tirge Cost LLw Fleq, (3122544)
066360041
1,34440065
0,512:02277
14400221
0
0513600115
0917400274
0.766900067
11P:00076
EvoCantrol (Pull State)
125
23.36011
142:90029
20.4590.055
77 77.360.451
9,325000085
6:3000003
31.4:1 34
5774:009077
46250,3366
25 226
EvoCanted (Residual
112:0:0999
EveCunteal (Targe Propein.)
14550.103
19.5:0499
639:0000579
31.560225
5712:0233
EvsCantedl (Target)
114.0.0776
22.9.0.156
14.5.00
19.410.467
76.21.0319
8.33100:19
7.000..000
30711:22
$.610.119
4.551000235
24.00.79
EvaCanted (Learnd Gains)
129:0.100
25, 1311200669
13.321 06
20.3:0.517
76.8.0.173
9.992000033
2.13:0.0462
21/411.52
5.4400
4.11100625
25.120.54
EroCanted Positio)
12.400.12
22330.162
20.5:0:499
73.2607733
8.99:900537
6.00
57260.151
459:200536
25.7:000029
J.8. Learning Curves for All Baselines
We provide the learning curves for all environments presented in the main table of results in the paper, that of Table 3.
Specifically, we provide these plots in Figures 5 to 14. We observe that EvoControl on average consistently outperforms the non-EvoControl baselines and achieves a higher evaluation return.
2500
2000
<
1500
1000
0.0
0.2
0.4 p (Policy Steps)
0.6
0.8
1.0
1e6
Direct Torque Cont.
Fixed Cont.
Fixed ConL.
EyeControl (Target + Proprio.)
EveControl (Leamed Gains)
EvoControl
Low Freq. (312)
PD Position
PD Position Delta
(Target)
Direct Torque Cont.
Fixed Cont.
EvoControl (Full State)
EvoControl
EvoControl (Delta Position)
High Freq. (500Hz)
PD Int. Velocity
(Residual State)
Figure 5: Evaluation return It versus policy steps on Ant, for main table of results Table 3.
44

---

EvoControl: Multi- Freqi-Frequenc Bi-Level Control for High-Prequency Continuous Control
3000
2000
R
1000
0.0
0.2
0.4 p (Policy Steps)
0.6
0.8
1.0
1e6
Direct Terque Cont.
Fixed Cont.
Fixed Cont.
EyoControl (Target + Proprio.)
EvoControl (Learned Cains)
EvoControl
Low Freq. (31.25 (31.25H2)
PD Position
PD Position Delta
(Target)
Direct Torque Cont.
Fixed Cont.
EvoControl (Full State)
EvoControl
EvoControl (Delta Position)
High Freq. (500Hz)
PD Int. Velocity
(Residual State)
Figure 6: Evaluation return At versus policy steps on Halfcheetah, for main table of results Table 3.
3000
2000
1000
0.0
0.2
0.4 P (Policy Steps)
0.6
0.8
1.0
1e6
Direct Torque Cont.
Cont.
EveControl (Target Proprio.)
EveControl (Learned Cains)
EvaControl
Low Freq. (31 (5z))Hz)
PD Position
PD Position Delta
(Target)
Direct Torque Cont.
Fixed Cont.
EvoControl (Full State)
EvoControl (Resitual State)
EveControl (Delta Position)
High Freq. (500Hz)
PD Int. Velocity
Figure Evaluation return R versus policy steps on Hopper, main table results Table
300000
200000
100000
0.0
0.2
0.4
0.6
0.8
1.0
p (Policy Steps)
1e6
Direct Torque Cont.
Fixed Cont.
Fixed Cont. Position Delta
EyoControl (larget Proprio.)
Eontrnl (Leameed Gains)
EvoControl (Target)
Freq. (31.25Hz)
PD Position
Direct Torque Cont.
Fixed Cont.
EvaContral (Full State)
EveControl
EvoControl (Delta Position)
High Freq. (500Hz)
PD Tnt. Velocity
(Residual State)
Figure 8: Evaluation return R policy steps Humanoid Standup, for main table of results Table
45

---

EvoControl: Multi- Multi-Frequene, Frequency Bi-Level Control for High-Prequency Continuous Control
8000
6000
<
4000
2000
0.0
0.2
0.4 p (Policy Steps)
0.6
0.8
1.0 1e6
Direct Terque Cont.
Fixed Cont.
Fixed Cont.
EyoControl (Target + Proprio.)
EvoControl (Learned Cains)
EvoControl
Low Freq. (31.25 (11225H2)
PD Position
PD Position Delta
(Target)
Direct Torque Cont.
Fixed Cont.
EvoControl (Full State)
EvoControl
Evo (Della Position)
High Freq. (500Hl2)
PD Int. Velocity
(Residual State)
Figure 9: Evaluation return R versus policy steps Inverted Double Pendulum, for main table of results Table 3.
1000
800
600
400
200
0.0
0.2
0.4
0.6
0.8
1.0
p (Policy Steps)
1e6
Direct Torque Cont.
Cont.
EveControl (Target Proprio.)
EveControl (Learned Cains)
EvaControl
Low Freq. (31 25H2)
PD Position
PD Position Delta
(Target)
Direct Torque Cont.
Fixed Cont.
EvoControl (Full State)
EvoControl (Residual State)
EveControl (Delta Position)
High Freq. (500Hz)
PD Int. Velocity
Figure 10: Evaluation return R versus policy steps on Inverted Pendulum, for main table results Table 3.
-1500
-200
-2500
-3000
0.0
0.2
0.4
0.6
0.8
1.0 1e6
p (Policy Steps)
Direct Torque Cont.
Fixed Cont.
Fixed Cont.
EvoControl (larget Proprio.)
Control (Leamned Gains)
EvoControl (Target)
Freq. (31.25Hz)
PD Position
Position Delta
Direct Torque Cont.
Fixed Cont.
EvoContral (Full State)
EveControl
EvoContol (Delta Position)
High Freq. (500Hz)
PD Tnt. Velocity
(Residual State)
Figure 11: Evaluation return R o policy steps Pusher, for main table of results Table 3.
46

---

EvoControl: Multi- Freqi-Frequenc Bi-Level Control for High-Prequency Continuous Control
-50
-100
150
200
-50
0.0
0.2
0.4 p (Policy Steps)
0.6
0.8
1.0
1e6
Direct Terque Cont.
Fixed Cont.
Fixed Cont.
EvoControl (Target Proprio.)
EvoControl (Learned Cains)
EvoControl
Low Freq. (31.25H2)
PD Position
PD Position Delta
(Target)
Direct Torque Cont.
Fixed Cont.
EvoControl (Full State)
EyoControl (Residual State)
EvoControl (Delta Position)
High Freq. (500Hz))
PD Int. Velocity
Figure 12: Evaluation return It versus policy steps on Reacher, for main table of results Table 3.
20
-40
60
80
0.0
0.2
0.4
0.6
0.8
1.0
P (Policy Steps)
1e6
Direct Torque Cont.
Cont.
EveControl (Targel + Proprio.)
EveControl (Learned Cains)
EvaControl
Low Freq. (31 22H))
PD Position
PD Position Delta
(Targel)
Torque
EvoContral (Full State)
EvoControl
EveControl (Delta Position)
High Freq. (500Hz)
PD Int. Velocity
(Residual State)
Figure 13: Evaluation return R versus & policy steps on Reacher ID, for main table results Table 3.
2000
1000
0.0
0.2
0.4
0.6
0.8
1.0 1e6
p (Policy Steps)
Direct Torque Cont.
Pixed Cont.
Fixed Cont.
EvoControl (larget Proprio.)
Evntrnl (Leameed Gains)
EvoControl (Target)
Freq. (31.25Hz)
PD Position
Position Delta
Direct Torque Cont.
Fixed Cont.
EvoContral (Full State)
EveControl
EvoContol (Delta Position)
High Freq. (500Hz)
Tnt. Velocity
(Residual State)
Figure 14: Evaluation return R p policy steps Walker2D, for main table of results Table 3.
47

---

EvoControl: Multi-Frequenge Bi-Level Control for High-FFrqqunnc Continuous Control
J.9. Ablation: Training High-Level Policy with Evolution Strategies
In the following we perform an ablation whereby we train the high-level policy with Evolution Strategies (ES) rather than PPO. We provide two variations: first, using ES to train the high-level policy with the same fixed low-level controllers for all non-EvoControl baselines (Table 30), and second, within EvoControl modifying training the high-level policy with ES instead of PPO.
First, we replace the high-level PPO policy, keeping the same high-level policy architecture and network as originally used in the main paper, and instead using the same ES as was used to train the low-level policy. For this we provide two versions, A) that of training the high-level policy with the same number of high-level p steps as the core baselines presented in the paper, with 1M p steps- labelled ES (IM) p steps). Specifically, this uses ssrrollouss = 1, K 88 es_poppsizz 256, es_sut as_sub__g _uu___enrrattons = 8. Second, B) training the high-level policy with the same number of environment steps as was used to train the low-level policy within EvoControl-labellee ES (same as EvoControl low-level). Specifically this uses more low-level environment steps than A), with es_ rollouts = 16, K = 8,, e,e_pop__ ppo_sizee = 512, es_ es_sub saab_ggnerations = 8. The results are tabulated alongside the original main table of results for ease of comparison in Table 30. We observe on average that EvoControl still outperforms the competitive baselines, and on average achieves a higher normalized evaluation return R than the baselines of fixed-controllers and direct torque control.
Table 30:
Ablation. Training the high-level policy with ES instead of PPO. Normalized evaluation return R. for the henchmark methods, across each environment. EvoControl on average achieves a higher normalized evaluation return than the baselines of fixed controllers and direct lorque control. Results are averaged over 384 random seeds, with 1 indicating 95% confidence intervals. Returns normalized a 0-100 scale, where represents a random policy, and 100 represents the highest reward achieved by Non-EvoControl baseline in each environment, Scores bolded are greater than 100.
Meched Nami
Lowied 3
Hkpper
Humandd
Hanadd Studlap
Incrted Double Peakikes
Incred Trudiiis
Praber
Readerr
Jeather
WalkrZD
ID
X-1
RCT
WCT
701
X0
X-1
X-1
X1
X.
Gue 1Dla: Valcky
PP0
F0 le: lakeety
2r6-151
2-54600977
75140504
1666155
D-0
197-1:55
56163
640
040
05-7-277
Cos Kunkes
Jasken
08383
Aiwd Boga Cos. High Fag (5085))
7P0
Disse Treqa
06.0
18:0.53
038343.55
35745357
0-0
040
43565555
046
4534634
0.0
Dinr Bege Owt Low thay 131 2900
1917
Dovel Tiege
606300
10011134 1:34
71609154
10
5011333
00:300111
1002 1352135555
15:08124
36 34
0181554
11
EeaDard Priqcic)
760
309-35.1
165-7.19
159
353
IveDbind (Tarpt)
1910
381:55
38
1151349
EraCaard (Leamed Galao
7PC
BS
366-184
113-183
366
159-151
137
116114
1661245
196.115
Earlbrdd (ldb Paikken
190
142-277
185-344
119:00
185 185-544
553521
9819027
181+ ML
teps)
skps
Albavos Fue. -D ka Weedyy
(lM
70 tc. Nakeet
125-2331
1756285
1561030
126.321
5521191
41200010
Abbave Finl/Ow Ovel Harder
kanken
Doaet Trege
000500
03303
00569
05991
AlMaNs Diwi Suqur Coc- High Treg (000
ES (IM) supa
Dose Tiea
254-221
13518.59
1171115
159113
100-128-45
344/030
751.0036
1961617
JANANI Dian Sequr Cop- Low Thea (11 2313
ES (1M5) supa
Dose Tiqq
195.311
13918.506
155:0340
106 1155
3001224
742-0200
105:3115
1841125
seper
Pon!
Alla'os Fuod -D ka Vilddy
(irnrrandd kanken
70 te. Nakecty
12771197
67562855
90.00
172.0/49
970.122
5521131
2561119
91.201115
567.0030
1481255
Akhav Fied/ (oel Randew
(hoei Tlage
00000
38448
03-35
00H00
00-00
00-00
00403
00159
08488
00403
00-00
Alaanis Oben Seqar Cooe- -igh theg g000a:
25 (BroDeand acaro siga
Diss Theqa
295.3305
13518.99
1171115
155-1
1661891
344JAR
1461837
169139
AWaan Dlan Sequr Cony- LLw the (31 (31235)
25 (brelbeand acans sigoo
Diss Ttrqu 223.1.56
13918.56
15000574
11918334
30:1244
742-0300
185:3011
1881237
Second, within EvoControl we perform the ablation of training the high-level policy with ES instead of PPO, using the same high-level policy architecture as used in the main paper. Again, we provide two versions, A) tha of training the A) that of training the high-level policy with the same number of high-level p steps as the core baselines presented in the paper, with IM p labelled ES (IM p steps). Specifically, this uses rooloutstts = 1, K 8, es_pop_sizee sip_size 256, es ss__u____g__ ll___uu__ee_errttee generations = 8. Second, B) training the high-level policy with the same number of environment steps as was used to train the low-level policy within more low-level environment steps than A), with es_ es_rollouts-- = 16, K = 8, es_pop_size = 512, e8 ses__ubbgeeeertttt The results are tabulated alongside the original main table of results for ease of comparison in Table 31. Crucially we observe that on average EvoControl using PPO to train the high-level outperforms (on average achieves a higher normalized evaluation return R) training the high-level with F.S, confirming the main framework and EvoControl method presented in the paper, and the advantages of the unique combination of high-level PPO learned policy with ES-lende low-level policy. Furthermore, we also observe that on average these variations of EvoControl outperform the respective con-EvoCottool baselines of credd-oontrollers and direct torque control.
Ciloonttoo-aablllle ES (same as EvoControl low-level). Specifically this uses
= 8.
48

---

EvoControl: Multi-Frequenee Bi-Level Control for High-Frequency Continuous Control
Table 31: Ablation. Training the high-level policy with ES instead of PPO for EvoControl. Normalized evaluation return R. for the benchmark methods, across each environment. EvoControl average achieves higher normalized evaluation return than the baselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 0-100 scale, where O represents a random policy, and 100 represents the highest reward achieved by a non-Evo non-EvoControl baseline in each environment, Scores bolded are greater than 100.
R1
x-
R.1
Cree 7D Roddoo
190
Pn1 Poction
10016.55
63
199
131-13
9941000
8034-.47
100-3.77
854399
Cree 7D Redtoo Dilts
191
PD Pootion Det
10:13
25110050
51:11.57
Im 50
01.0
152.76
565110219
Cont Voleiy
1P0
Velaity
2.5590:967
0.0
40:2:1.35
0m0
0.0
05772277
Ronfict
00:30
00:00
00:00
0.000.
00:00
0,0000
chaitt
D200
EroCastré (hat Hand
190
31.:13.2
14.3
274::206
123:19
136:184
101:9200
106:00
362:12.5
114:7.51
386::275
133
FooCattdd (Beittal State)
190
1921
162 6.31
5.54
176
312 165
100-0
355 91.5
266 57.4
FroCattnd (Target Troprie)
190
319
168
155
7339
150
99.11119
353
17N
EnoCateed (Teggt)
190
EroCatted (Lareed Gaieo)
160
2M::143
09.5+248
ANatio: ErwCanted (Pill Ntate
1s (IM, papp
151: 02.9
134 1.4
115:313
134:181
WA111
328:96.2
55.4.164
-2.11563
AMulia: Enwisnted (Ronhial State)
15 (15 aqu)
136:135
9642213
136:13
115:365
0771389
17.525.11
232:22.3
5233115
2963212
AMalion: FroErated Proprie)
131-13.9
151
18441116
121 26.7
187-235
8:457-
23-186
246-76.3
530-171
234-176
126
EnClated
135:1.57
6344405
AMatian: EnCanted (Leannal
pgg
174:36.9
6722199
46.72233
2625204
ANatio: Enoianted (Pill Statel
1X [DoCeenn
176.66.9
163 12.4
914.156
24.
388
95,11951
334:55,9
15.1.16
157126.1
153. 153.4415
AMulia: Enwiatted (Rodital State)
E E
169153.8
154
90.118.6
131-4441
142.55
(3311664
15811994
389:361
52332003
-3011001
142:118
AMalion Fnointted
Progrie)
153-13
122-26.4
102-233
62:74158
0356557
391-164
5361123
27.3-524
122-26
AMalion Fnoistted
153161
63:5-7311
130 23.9
096-25.1
68-347
20:4115
113:713
511-187
18:1-54
116-113
EnClated
5388139
AMatbar: EnDiateed (Ddia Paelbon
35
172:97.5
J.10. Rollout Trajectory Plots of High-level Action for Baselines
In the following we plot rollout trajectories, including the high-level latent action O₂ over one evaluation episode in the Reacher ID environment for all baselines. We take the trained policies, as trained in the main table of results in the paper (Table 3), and evaluate them for one random seed to produce the rollout trajectory plot. To facilitate simpler comparison, we use the same random seed across the baseline trajectory plots. Moreover, we use the Reacher ID environment straightforward to plot and understand what an optimal policy should do. In this case the initial reacher arm starts in random position (as defined on a circle go E (-H,, r)), with random velocity, and its goal is to move the arm to a randomly sampled goal location (goal Ieoal E (- (f,,,)))in in the trajectory plots we plot the goal location with the red line, which is constant throughout the episode. Therefore an optimal policy is one that moves the arm to the goal location, quickly and keeps it there to maximize reward, where the reward is defined as -||xy [|y__iip__ _tt___poo__ of arm arm(go) - xy__ip _tt__ppo____ of_of______ arm (ugaggaa) ll.. We provide the plots for each respective baseline in Figures 15 to 25. We observe that the EvoControl variants on average consistently outperform the other con-EvoControl baselines and achieve higher evaluation return R--hhich is also given on each plot.
is
Fixed Cont. - PD Position
0.3
0
0.0
0
-1.3
-2.0
0.0
-0022
Re 21.99
0.00
0.25
0.50
0.75
1.00
1.23
1.50
1.73
2.00
t
Figure 15: Evaluation Trajectory Rollout for Reacher ID, for baseline Fixed Cont. - PD Position. Environment runs at 500Hz, with an 1,000 low-level environment steps, corresponding to a episode duration of 2.0 seconds. Where a is the high-level policy latent action. qo is the reacher arm's angle in radians, with the eine indicating the random goal good for the episode, r: the instantaneous reward and R the total return for the episode.
49

---

EvoControl: Multi- Multi-Frequency Bi-Level Control for High-Prequency Continuous Control
Cont.
PD Position Delta
1.0
0.5
Geal
-1.3
-2.0
-0.1
-.2
-0.3
t
Figure 16: Evaluation Trajectory Rollout for Reacher 1D, for bascline Fixed Cont. PD Position Delta. Environment runs at 500Hz, with 1,000 low-level environment steps, corresponding episode duration of 2.0 seconds. Where og is the high-level policy latent action, go the reacher 's angle radians, with the red-line indicating the random goal Qoal for the episode, 1' the instantancous reward and the total return the episode.
Fixed Cont. PD Velocity
0.25
8-13
-2.0
-0.2
-0.3
Ri 23.90)
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Evaluation Trajectory Rollout for Reacher ID, for haseline Fixed Cont. PD Int. Velocity, Environment runs 500Hz, with an 1.000 low-level environment steps, corresponding to episode duration of 2.0 seconds. Where as is the high-level policy latent action. is the reacher arm's 's angle in radians. with the red-line indicating the random goal quai for the episode, rı the instantaneous reward and Rt the total return for the episode.
Figure 17:
Direct Torque Cont. High Freq. (500Hz)
1
0
8 1.5
-2.0
-0.0200
-21.223
-0.0225
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
Figure 18: Evaluation Trajectory Rollout for Reacher ID. for baseline Direct Torque Cont. High Freq. (500Hz). Environment runs 500Hz, with an 1,000 low-level enviroument steps, corresponding to episode duration seconds. Where as the high-level policy p latent action, go is reacher arm's angle in radians, with red-line indicating the random goal goot the episode, re instantancous reward and R the total return for the episode.
50

---

EvoControl: Multi- Frequ-Fre Bi-Level Control for High-Prequency Continuous Control
Torque Low Freq. (31.25H2)
60
--.5
Geal f
-2.0
0.0
0.2
n= 0.08
Evaluation Trajectory Rollout for Reacher ID, for baseline Direct Torque Cont. Low Freq. (31.25Hz) Environment runs 500Hz, with an 1,000 low-level environment steps, corresponding to episode duration of 2.0 seconds. Where dą the high-level policy p latent action, go is the reacher arm's angle in radians, with red-line indicating the random goal Zyal for the episode, rt the instantaneous reward and I total return for the episode.
Figure 19:
EvoControl (Full State)
2.5
0.0
--.5
-2.0
0.0
-022
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Figure 20: Evaluation Trajectory Rollout for Reacher ID, for haseline AvoControl (Full State). Environment runs at 500Hz, with an 1,000 low-level environment steps, corresponding to a episode duration 2.0 seconds. Where as is the high-level policy latent action. is the reacher arm's angle in radians, with the red-line indicating the random goal quai for the episode, the instantaneous reward and It the total return for the episode.
EveControl (Residual State)
2.5
0.0
-1
Gool
-2
0.0
0.2
0.00
0.25
0.50
0.75
1.00
1.23
1.50
1.75
2.00
Figure 21: Evaluation Trajectory Rollout for Reacher ID, for baseline EvoControl (Residual State), Environment 500Hz, with 1,000 low-level environment steps, corresponding a episode duration 2.0 secouds. Where as the high-level policy latent action. do is the reacher arm's angle radians, with the red-line indicating the random goal gool episode, r, the instantancous reward R the total return for the episode.
51

---

EvoControl: Multi-Frequeee Bi-Level Control for High-Prequency Continuous Control
EvoControl (Target + Proprio.)
2.5
0.0
-1
Geal n
-2
0.0
0.2
n= 0.544
Figure 22: Evaluation Trajectory Rollout for Reacher ID, for baseline EvoControl (Target Proprio.). Environment runs at 500Hz, with an 1,000 low-level environment steps, corresponding to episode duration of 2.0 seconds. Where is the high-level policy latent action, go the reacher 's angle radians, with the red-line indicating the random goal Qsal for the episode, 1x the instantancous reward and the total return episode.
EvoCentrol (TTrgett)
0.0
-1
-2
0.0
--202
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Figure 23: Evaluation Trajectory Rollout for Reacher ID, for baseline EvoControl (Target). Environment runs at 500Hz, with an 1,000 low-level environment steps, corresponding to episode duration of 2.0 seconds. Where aş is the high-level policy P latent action, go the reacher arm's 'ss angle in radians, with the red-line indicating the random goal qyya for the episode. the instantaneous reward and I? the total return for the episode.
EveControl (Learned Gains)
2.5
6 0.0
-1
8
Gmal
-2
0.0
0-2
0.00
0.25
0.50
0.75
1.00
1.23
1.50
1.75
2.00
Figure 24: Evaluation Trajectory Rollout for Reacher ID. for baseline EvoControl (Learned Gains). Environment 500Hz, with an 1,000 low-level environment steps, corresponding to episode duration seconds. Where as the high-level policy p latent action, do the reacher arm's 's angle radians, with red-line indicating the random goal gool the episode, r; the instantaneous reward and R the total return for the episode.
52

---

EvoControl: Multi- Multi-Frequency Bi-Level Control for High-Prequency Continuous Control
EvoControl (Della Position)
--1.
0oll
-2.0
0.0
C-0.2
-063
0.00
0.25
0.50
0.75
1.00
1.23
1.50
1.75
2.00
Evaluation Trajectory Rollout for Reacher ID, for baseline EvoControl (Delta Position). Environment 500Hz, with 1,000 low-level environment steps, corresponding episode duration of 2.0 seconds. Where ag the high-level policy latent action, episode, v; the instantaneous reward
Figure 25:
do is the reacher arm's angle radians, with the red-line indicating the random goal Ogoll R the total return for the episode.
J.11. A Ablation: Removing Communication Between the Layers in EvoControl
We performed a further ablation, which tests the hypothesis of if within EvoControl, whether there is useful communication between the high-level and low-level policy. Specifically for the EvoControl variants that considered, in some variations the low-level policy receives only restricted observation (just the joint positions of the robot, and not the random goal location one exists), compared to receiving the full observation (which includes any random goal location if one exists for that environment).
Specifically, we consider two variants of EvoControl, where the low-level policy receives the full observation (EvoControl (Full State)), and where the low-level policy only receives a restricted observation without any goal location -ertiiitnnnnssstatttininn effective communication from the high-level to the low-level (EvoControl (Target)). We compare these on the Reacher ID task, visualizing the rollouts for random high-level policy, and a zero high-level policy, observed in Figures 26 and 27. We observe that for both variations after the standard EvoControl training, removing the communication (by making the high-level policy either random policy or null policy) significantly reduces the performance (return), and leads to an unstable low-level policy, even when the low-level policy receives the full observation (st). Intuitively, it could be the case that the low-level policy learns a form of a PD controller, and the high-level policy, as trained initially with PD position controller, could converge to treat the low-level policy as a form of PD controller.
53

---

EvoControl: Multi-Frequene Frequency Bi-Level Control for High-Preqqency Continuous Control
EvoControl (Full State)
2.3
8 0.0
-1.3
Cool
-2.0
0.0
-.2
A- 6.70
0.00
0.25
0.75
1,00
1.23
1.50
1.75
2.00
EvoControl (Full State) Ablation; Random High-level policy
0
-1
2.3
0.0
2.5
-2
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
EvoControl Ablation; Zero High-level policy p
(Full State)
0.05
2.5
-2
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Figure 26: Evaluation Trajectory Rollout for Reacher ID, for baseline EvoControl (Full State), with ablation of random high-level and a null (zero action) policy. Here the observation for the low-level policy is 8;, policy receiving the full observation still relies the communication from the high-level latent action ag, and without it, the return significantly reduces. Environment seconds. Where as is high-level policy latent action, do the reacher arm's angle radians, with the red-line indicating random goal queat for the episode, rt the instantaneous reward and Ii the total return for the episode.
qe, t/T. We observe that even with the low-level
500Hz, with 1,000 low-level environment steps, corresponding
episode duration of 2.0
54

---

EvoControl: Multi- Frequency Bi-Level Control for High-Preqqency Continuous Control
EveControl
(Target)
2.5
8
0.0
Coal 9
-2
0.0
--2.2
R- -0.34
0.00
0.25
0.75
1,00
1.23
1.50
1.75
2.00
EvoControl (Target)- Ablation: Random High-level policy
0
-1
-2.
07.523
--1
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
EvoControl (Target) Ablation; Zero High-level policy p
0.05
-0.5
8-2
-3
C-1
0.25
0.50
0.75
1.00
1.50
1.75
2.00
Figure 27: Evaluation Trajectory Rollout for Reacher ID, for baseline EvoControl (Target). with ablation of a random high-level and a null (zero action) policy. Here the ohservation for the low-level policy is ax It, 9t, t/T. We observe that the low-level policy relies on the communication from the high-level latent action and without it, the return significantly reduces. Environment runs at 500Hz, with an 1,000 low-level environment steps, corresponding episode duration 2.0 seconds. Where as the high-level policy latent action, go is the reacher arm's angle radians, with the red-line indicating the random goal quat episode, rt. the instantaneous reward and R the total return for the episode.
55

---

EvoControl: Fretu--requency Bi-Level Control for High-Freqqenny Continuous Control
J.12. Ablation on Varying Fixed Values of the Annealing Parameter
First, we clarify that in EvoControl, the annealing parameter a is gradually reduced from 1 to 0 during training, Instead, we analyze the impact of fixing to constant value throughout training.
By default, EvoControl anneals a over time, balancing control between the high-level and low-level controllers: O means full control by the high-level policy, while = 1 means the low-level controller (initialized as a PD controller) has full control. We present this analysis in Table 32.
Table 32:
Ablation of EvoControl with fixed parameter. Normalized evaluation return R. for the benchmark methods, across the Reacher ID environment. EvaControl average achieves higher normalized evaluation return than the haselines of fixed controllers and direct torque control. Results are averaged over 384 random seeds, with + indicating 95% confidence intervals. Returns are normalized to a 1--100 scale, where represents a random policy, and 100 represents the highest reward achieved by Non-EvoConttol baseline in each environment. Scores bolded are greater than 100.
Reaher ID (a= (.0)
Reacher ID (o - 0.13 Reacher ID (o - 0.2)
Reacher ID (a 0.4)
Reucher ID (a 0.6)
Realher ID (a 0.8)
Reacher ID (a -1.0)
Same PPO
R1
Fized Cost. Position
35::293
$5:22.3
55::2.93
95::2.90
85:62.93
85::2.3
$5:22.33
Plaed Clsar PD Positio Delta
15/26766
15,247.6
15.28/7.6
15.26:76
1522:76
15:2:766
15,247766
Fiond
Velecity
0+0
0+00
0+-0
0+00
Pired Conr. Random
0,03000
0,00000
0.0:0000
00:000
00::000
00:0000
0,06000
Direct Torgue Cosr. High Frog. (800))))
45,316674
14.446674
45 Se-5
45 3-1,74
45.366.7
45.3-6674
45, 44.346674
Divect Toquur Coni -Low Frog (31 25H4)
10261.94
100:1.94
100:1.94
10001104
10061.94
100:1.94
ExoCantral (Full Stade)
105::1.25
106:0.781
106:00.765
166:1.31
805:08822
166-1
89.762.00 70
EvoCantral (Residual
105:1.03
105:0.916
105:10,303
105::0825
104::1.0
106:0.754
89.722.79
EvoCoatrol (Target Proprio.)
105:0.9447
106:00493
105:1.17
106:116
106:0.844
195:00.3
89.722279
ExoCantrol (Target)
104:0,943
165-0..94
1.5-11112
104:1.29
104:142
105:00935
89.75:2.99
ExoCoutrol Gains)
108:11.66
105:1.65
105::0.309
102::1.19
91.7:228
89.722279
ExoCCnntrol (Delta Position)
10L::2.5
185::126
162:1.55
100:1.85
109:2.84
98.88:2441
24.187.45
EvoControl consistently outperforms baselines across most a values, demonstrating the effectiveness of our hierarchical approach. Performance degrades for some EvoControl variants at a 1.0, indicating the importance of high-level policy guidance. This aligns with our annealing strategy, gradually reducing the low-level controller's 's influence the high-level policy learns. The low-frequency torque control is competitive, but EvoControl achieves higher rewards, especially for lower values, showcasing the benefits of learning the low-level control.
J.13. PD Position Delta Action Scaling Sweep
We performed an additional experimental action scaling sweep of the baseline Fixed Controller - PD Position Delta. Specifically this baseline has a apper-prrmmter of the action delta scaling parameter Se. We sweeped over ó. for the Reacher 1D environment and present the results in Table 33. We observe that a larger &e does improve the baseline's performance, however it still underperforms compared to EvoControl.
Table 33: Fixed Controller - PD Position Delta, action detla scaling parameter & sweep. Normalized evaluation return R for the baselines across the Reacher ID environment. We use the same normalization the main table of results in the paper, that of Table 3.
Goaterr 66. i 0033
Rtocher iD 0.3)
Rtacher 1D
02)
Readher S 0.0
Itoccer iD 0.00
Reacher ID14,
0.80 Reader 1l 1.00
Rtacher 16 2.0)
Sumee I9D ligh dip
R:1
R2
R1
R-
R1
R1
Floed Cort. 10 Pestian Uoto
25.6
35528997
$7.31..99
26:4.12
$1533664
832:3344
13.13343
$4.52328
56

---

EvoControl: Multi-Freqeeccy Bi-Level Control for High-Prequency Continuous Control
J.14. Safety Critical HalfCheetah
freeffreeeecyy control is especially valuable in safety-critical locomotion, where an agent must react instantaneously unforeseen contacts. In the Safery- CrietcCCitticl HalfCheetah task (Appendix E.4) we insert a blocking capsule in 25% of episodes and expose only the measured contact force to the policy-the the observation vector is of dimensionality 19. A successful controller must therefore (i) accelerate forward to maximise progress, yet (ii) immediately reverse direction upon detecting a collision in order to avoid the large penalty of 200 (c.f. Eq. equation 3). Such behaviour demands rapid, reflex-like adjustments that nower-frequent policies struggle to produce, thereby providing an empirical test-bed for Proposition 2.1. This enviormeent is illustrated in Figure 28.
Table 34 reports the normalised evaluation return R (t)-scaled to [0, 100] where 0 is random policy and 100 is the best non-EvoControl baseline- consistently outperform both fixed low-level controllers and direct torque control, even when the latter operates at 500 Hz. Notably, EvoControl (Full State) attains 120 + 13.9, exceeding the best baseline by more than 20% despite using the same high-level PPO algorithm. These results corroborate our hypothesis that the ability to adapt low-level actions online at high frequency is crucial for safe and efficient locomotion in un un-modelled environments.
over 384 random seeds with 95% confidence intervals. EvoControl variants
Table 34: Normalised evaluation return R on the Safety-Critical HalfCheetah. EvoControl achieves significantly higher returns than both fixed high-frequency tracking controllers and direct torque control. Bold scores exceed the best baseline (100).
Same PPO high-level algorithm o with
Safety-Critical HalfCheetah R. (t)
Fixed Controller PD Position
84.7 1,68
Fixed Controller PD Position Delta
0.0
Fixed Controller Random
0.0
Direct Torque Control High Freq. (500 Hz)
100 1.42
Direct Torque Control - Low Freq. (31.25 Hz)
0.0
EvoControl (Full State)
120 13.9
EvoControl (Residual State)
105 5.84
EvoControl (Target Proprio.)
117 34.9
EvoControl (Target)
117 5.33
(a) No Collision
(b) Blocking Wall
Figure 28: Safety Critical Halfcheetah Environment-
halfcheetah enviormment (a), with a blocking wall that appears 25% of the episodes (b), and only contact force with wall can be observed, blocking the chectahs goal path, and incurring negative reward for collision with the wall. Therefore good policy in this environment is move forwards with the halfcheetah and if the wall collision detected immediately retreat, otherwise continute forwards, therefore exemplifies the need policies that take actions higher-frequency to achieve higher reward, un-modelled safety critical applications.
57

---

EvoControl: Multi-Frequeng, Bi-Level Control for High-Prequency Continuous Control
J.15. Comprehensive Baseline Comparison
To contextualise the gains of EvOCONTROL benchmark it against a wide suite of strong controllers, including both fixed high-froet fegh-freeuenyy tracking controllers and state-of-the art model-free RL agents. We additionally train Soft Actor -riticc (SAC) and Proximal Policy Optimisation (PPO) with publicly available Brax" implementations at high frequency (500 Hz) and the high-level low frequency (31.25 Hz). Pre-trained policies from standard MuloCo tasks cannot be used directly because our tasks require an order-of-magnitude faster actuation; instead, we retrain all methods from scratch under identical hyperparameters. We tabulate these results Table 35.
Normalisation and statisties. Returns are undiscounted and linearly normalised [0, 100) such that corresponds to random policy and 100 to the best non-EvoCONTROL baseline in each environment. All numbers are means over 384 random seeds; = indicates the 95% confidence interval.
Table 35: Normalised evaluation return R for all baselines and EvoCONTROL on the eight benchmark environments. Bold values exceed the best non-EvoCONTROL bascline (R > 100).
Same levhllevel alg p with
Halfcheetah
Hopper
Inventd Double Pendulom
Inverted Pendulum
Reacher
Walker2D
Low-Lovel Policy
ID
1
R
R t
R
Fised Cont. Position
10016.56
61.2:104411
88.121.18
99.9:0003
100 2ire:2.6c-115
85.24:2.87
61.120.51
Flxdd Conz. PT) Position Delta
2.4+1.911
2.76+DD.0888
96.2+1.30
53.8+1.57
100--2.06e- 15
40.943323
15.2+77.6
72.7+0.19
Fixed Cont. PD Ins. Velecity
3.59
2,46:10.0932
71.8:0.87
49.7
86.51.2
0,0:0.0
01:0
69.312.06
Fixed Comt.-P7 Position & Kp
3.55#2.54
16.7=0.151
100+:1.00
97.5=0.751
100:22.86c 15
50.7=3.9
81.8:44111
100:+0.24
Fiord Conz. Random
0.0+0.00
0.0:0.00
0.0+0.0
0.0::0.0
0.0++0000
0.0:00.0
0.0:00.0
0.0+0000
Dive Torque Conz. High Freq. (500Hz)
0+0
17.2+00.16
1.37+0.51
10,3-0,586
0++0
0.97+-5.72
45.3+-674
0.0+0000
Direct Torque Cont. LLo Freq. (31.25H12)
54.517.15
10011:21
69.2:10.62
80.61:2.56
100:0003311
53.0:1..35
1001:194
80.71:2116
EvoControl (Full State)
368::10.6
157:1.1
263:::.46
101:0.0487
100:2.886-15
114:009733
106:00993
163:13772
EvoControl (Residual State)
182+8.58
182+1.00
97+00.11
99.2-0.054
100+-2806-15
106+-1.29
104+1.19
165++2.1
EvoContral (Target Pruprio.)
319114.1
16811.41
16415.08
99.7110.0417
100
96.813..4
105100777
143:
EvoControl (Target)
293:13.2
162::1.58
2724:184
99 99.6220.3377
100=22866-15
112::0.785
105:0.78
151:22.4
EvoControl (Learned Gains)
2661:4.11
113:116
198::962
99.520009447
100=2.86e-15
116:00.47
105::121
158::3.63
RvoControl (Delta Position)
362+12.8
133+1.82
101+0.0334
100--1806-15
99.1+2.44
147+1.90
(Brux) Torque High Freq. (5001I2)
1188:264
33.6:06646
106+1.43
155:40.533
Direct Torque
Freq. (31.25Hz)
12.81:269
28.320.478
90.210003433
105122008
156100.41
PPO (Brax) Direct Torque Cont, High (500H2)
1,23+11311
18:2+0 346
81:+0.0255
62.11-4.13
104:1.54
114:2.67
PPO (Brax) Direct Torque Cont. Law Freq. (31,25112)
3.1411.11
18.2::0.46
81:10.0225
100.:0
100.00
62.114.13
10411.54
114::267
Across all eight environments EvoCONTROL. attains the highest average normalised return, frequently 50% above the strongest baseline. Crucially, even when sophisticated RL agents are granted the same 500 Hz control bandwidth, the population- based evolution of low-level reflexes in EvOCONTROL yields superior performance, underscoring the importance of closed-loop online adaptation at millisecond time-scales.
htttps://gi thub.
com com www..//oooooooooooooooooooooo 1e/oge/oggooaooooo
58

---

EvoControl: Multi-Frequene, Bi-Level Control for High-Prequency Continuous Control
K. Real-Worke Validation on Franka Emika Panda
validate EvoControl on a 7-DoF Franka Emika Panda robot equipped with a Robotiq 2F-85 gripper and a Robotiq FT-300 force-torque sensor, aiming to demonstrate two key aspects:
1. Zero-shot sim-to-real transfer tabletop manipulation task;
2. Reduced collision forces via feeuffre torque control, compared a simpler position- position-based controller.
Figure 29 shows the overall setup in both MuJoCo simulation (left) and the physical robot (right).
Figure 29: Real-World Robot Setup. Left: MuloCo scene of our 7-DoF Franka Emika Panda robot equipped with a Robotiq 2F-85 gripper and a Robotiq FT-300 force-torque sensor. Right: The corresponding real hardware. We deploy two sim-to-real tasks to evaluate EvoControl.
Figure 30: Block Collision and Bin-Opening Tasks. Left: The robot collides with a block while moving its end-effector downward, mcasuring contact forces. Right: Another example of applying high- gighfrreueeny torque control for soft bin-opening motion (qualitative demonstration).
Task and Setup. We evaluate two tasks to highlight both free-space tracking and collision handling:
59

---

EvoControl: Multi-Frequeeii Bi-Level Control for High-Frequency Continuous Control
1. Sim2Real Goal Tracking:
The robot must track randomly moving goal pose in free space. A shaped reward encourages precise positioning of the end-effector (tool center point, TCP), through reward to minimize the 1.2 norm distance of the TCP to the goal location.
2. Sim2Real Block Collision: The same policy is tested with an enforced collision by commanding the end-effector to move downward into block. We measure the average forces at the end-effector.
In both scenarios, the low-level controller operates at 200 Hz (torque control) while the high-level policy (PPO) runs at 10 Hz. We train each policy entirely in MuJoCo and deploy it zero-shot on the physical robot via the Franka Control Interface (FCI). The robot's built-in gravity compensation is enabled, allowing the learned controller to focus purely on torque-level corrections and collision handling. The same robot platform was also used and developed further in Zakka et al. (2025).
Baseline Controllers. We compare EvoControl (Full State) against two alternative low-level controllers, both paired with the same high-level PPO policy learning algorithm:
Fixed Controller- PD Position: A hand-tuned PD loop that tracks the high-level's joint-position commands.
Direct Torque Control (Hi-Freq): An end-to-end torque policy trained purely with RL at high frequency.
While direct-torque RL can in principle match EvoControl's frequency, it often suffers from poor exploration and long-
horizon instability, as shown in the main paper.
Metrics.
We measure:
Avg. Tracking Error (m): The distance between the goal pose and the end-effector over an episode.
Avg. Total Force (N): The mean force magnitude recorded at the wrist sensor, capturing the compliance stiffness of each approach.
Results. Table 36 presents the performance of each low-level controller across three conditions: (1) Sim-Goal Tracking (evaluation in simulation), (2) Rim2Real Goal Tracking (physical free-space tracking), and (3) Sim2Real Block Collision (intentional collision).
Accuracy vs. Compliance: EvoControl maintains comparable or better tracking accuracy than the fixed PD controller while reducing collision forces.
Zero-Shot Transfer: No additional fine-tuning was performed on real hardware; EvoControl exhibits stable behavior from the outset.
Table 36: Performance comparison of different low-level policies with a high-level PPO algorithm. EvoControl achieves superior tracking performance and lower force application compared to other methods. Direct torque control at high frequency fails to complete the task due to hitting joint limits. Bold values indicate the best performance in each metric.
Sim-Goal Tracking
Sim2Real Goal Tracking
SIm2Real Block Push
Same PPO high-level alg. with
Rt
Avg. Tracking Error (m) J.
Avg. Total Force (N) L.
Avg. Tracking Eroo (m) Avg. Total Force (N)
Fixed Cont. PD Position
13,564
0.076
4.159
0.1537
9.056
Direct Torque Cont, High Freq.
10,413
Not Finish (Hits joint limits)
Did Not Finish (Hits joint limits)
EvoControl (Full State)
24,213
0.0261
4.331
0.1524
8.169
Conclusion. These real-robot results confirm that EvoControl can: Transfer directly from MuloCo simulation to real hardware without additional tuning: Retain accuracy for free-space tracking while show initial insights into high-frequency torque controlled policies potentially having lower un-modelled interaction forces. Overall, this demonstrates EvoControl's practical viability for deployment to real robot setups to assist the future development of safe, adaptive, figh-frequency torque control in real-world robotic tasks.
60

---

EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
L. Limitations & Future Work
While EvoControl demonstrates promising results for high-frequency continuous control, several limitations and avenues for future research warrant exploration.
Still relies on the existence of a fixed-PI controller for the continuous- timeinuuus-time control task. Although we demonstrate robustmess to some degree of PD parameter misspecification (Table 6), the reliance on a PD controller as starting point poses a limitation. In domains where designing a suitable PD controller is challenging or impossible (e.g., systems with non-actuated joints, highly nonlinear dynamies, or discrete action spaces), applying EvoControl in its current form may be difficult. We do perform an ablation where we show that EvoControl can still learn performant policies without the existence of a fixed-PD controller Appendix J.5, however other approaches to stabilize and initialize policy learning are promising directions for future work.
EvoControl can require more computational complexity compared to only performing PPO, which can be readily parallelized in practice with modern accelerated compute platforms, and restricting EvoControl to use the same computational complexity, whilst still outperforming the baselines is also possible, Appendix J.4.
In addition, promising future directions include exploring more complex nested hierarchies, direct low-level to high-level information flow, and ensembles of policies.
M. Reproducibility Statement
In the following we outline all the sections where the reader can find full information to fully reproduce all the main results. We also clearly state the following of the assumptions of the method in Appendix G, experimental settings in Appendices to I, and the limitations of the work in Appendix L.
N. Common Questions and Discussion
N.1. Why Not Simply Pre-train with a PD Controller and Imitate?
One might consider using a PD position controller as a policy to collect rollout trajectories and then using imitation learning to train neural network to replicate behavior. However, this approach has limitations. The learned network would only be capable as the PD controller itself, inheriting its limitations in expressiveness and inability to learn complex, freeueryyyncy interaction behaviors. EvoControl, by directly optimizing the low-level policy with ES, aims surpass the capabilities of the initial PD controller and discover more nuanced and adaptive control strategies. Furthermore, imitation learning requires substantial amount of demonstration data, while EvoControl learns directly from the environment reward signal.
N.2. Relationship to Pulse- Width Modulation
The benefits of high-frequency control, highlighted by Proposition 2.1, share a conceptual similarity with Pulse-Width Modulation (PWM) in electrical engineering. In PWM, a high-frequency signal with varying pulse widths is used to effectively represent lower- twwerffeeuency analog signal. Similarly, in EvoControl, figh-freyeency actions generated by the low-level policy can represent and achieve the lower-frequency targets set by the high-level policy with greater precision fred-freuuency PD controller. While not direct analogy, this parallel highlights the ability of high-frequency signals enhance control and achieve desired outcomes more effectively.
and responsiveness compared
N.3. Discussion of Theoretical Analysis on Convergence of the Bi-Level Framework
While formal proof is beyond our current scope, we have designed EvoControl to promote stable learning and convergence through the following mechanisms:
Staged Training and Annealing
Alternating optimization of the high-level policy p (using PPO) and the low-level policy 3 (using evolutionary strategies) reduces non-stationarity.
61

---

EyoControl: Multi-Freqeecy Bi-Level Control for High-Prequency Continuous Control
The annealing parameter a gradually transitions 3 from fixed PD controller learned neural network, maintaining stability.
Convergence Properties of PPO and Evolutionary Strategies
PPO: Known for stable convergence due to its clipped surrogate objective (Schulman et al., 2017).
Evolutionary Strategies: Robust in high- high-dimensional, non-convex spaces, effectively optimizing policies without gradient information (Salimans et al., 2017).
Relation Hierarchical Reinforcement Learning
Our framework aligns with HRL methods such the Options framework (Sutton et al., 1999) and FeUdal Networks (Vezhnevets et al., 2017), which provide theoretical insights into hierarchical policies.
Nachum et al. (2019) offer theoretical guarantees for HRL methods, supporting potential convergence in hierarchical structures.
These design choices contribute to the stability and reliability of EvoControl, aligning with established reinforcement learning principles.
N.4. Comparison of EvoControl with Model Predictive Control (MPC)
EvoControl differs from MPC in key aspects:
Learning-Batt Based vs. Model-Based
EvoControl: Entirely leaming-based, using neural networks trained through interaction with the environment, without explicit system models.
MPC: Relies explicit mathematical models to predict future states and optimize control actions.
Policy Optimization Online Optimization
EvoControl: Policies are optimized during training and map states to actions without online optimization during execution.
MPC: Performs online optimization at each control step during execution.
Hierarchical Structure
EvoControl: Employs hierarchical policy with temporal abstraction, where high-level policy guides low-level policy at higher frequency.
MPC: Does inherently incorporate hierarchical policies with different operating frequencies.
Novelty and Significance
Integration of PPO and Evolutionary Strategies: Combining PPO for the high-level policy with evolutionary strategies for the low-level policy in a hierarchical setting is, to our knowledge, novel.
Transitioning from fixed PD controller to learned policy stabilizes training, feature not present in MPC.
Annealing Strategy:
Empirical Performance: Our experiments show superior performance in figh-frequency control tasks compared to standard RL methods and fixed-controller baselines.
62


Reviews:
[{"review_id": "gijqS2xuCb", "reviewer": "Reviewer_kbC8", "summary": "The paper introduces a bi-level policy and training method for high-frequency and continuous-time control. The bi-level policy consists of a high-level policy that operates at a low frequency and issues a latent action. The low-level policy decides the final action based on the environment state and the latent action. For training stability, the low-level policy is a convex combination of a ​PD controller and a neural network where the combination gradually shifts towards the NN policy during the training. The high-level policy is trained with PPO and the low-level policy is trained with an evolution algorithm. The paper shows this approach can outperform both direct torque learning and fixed PD controller approaches and introduce exploration and robustness advantages over them.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback, particularly their appreciation that all the claims are well supported and the paper does a tremendous job supporting the findings, with the idea being novel and interesting. Below, we address each concern:\n> fixing the total time and not the number of steps is more fair. If possible, including the results for this will be beneficial.\n\nThis is an excellent point, we agree. We already provide results for all baselines in **Appenidx J.4** (also shown here: https://imgur.com/a/CBzbEBO) where we fix the total time—here implemented by fixing the total number of low-level policy steps, which for all our high-frequency environments, that take actions at 500Hz, is a duration of 2 seconds. This ensures that the direct torque controller is evaluated in an identical physical time window to EvoControl. These new results still show EvoControl outperforming baselines, including the high-frequency direct-torque policy, which continues to struggle with long-horizon credit assignment at 500 Hz.\n\nWe will highlight these findings more prominently in the main paper’s Table 3 (as suggested) for camera-ready. This way, readers see both “fixed high-level steps” and “fixed total environment time” versions of the experiments.\n> Typos\n\nThank you, we have now fixed these.\n> Are there any experiments to show the proposed method has advantages over hierarchical RL methods with goal-reaching low-level policies, such as HIRO.\n\nYes, as HIRO closely resembles our *\"fixed-controllers\"* methods class (L192), where the low-level policies perform sub-goal state completion. Let us clarify that HIRO is an off-policy method that assigns state-based goals to a low-level policy, rewarded by sub-goal completion. However, as shown in Table 4, sub-goal attainment methods cannot handle the high-frequency reactions needed in safety-critical tasks. \n\nWe did use an existing implementation of HIRO (from the authors) on our high-frequency environments, however, we found that it was too unstable, and did not converge to a meaningful policy (showing random performance). We likely believe that HIRO, being an off-policy method, may require extensive hyper-parameter tuning for each environment and task to work correctly, and the authors within the HIRO paper discuss the instability of the method during learning.\n\nSimilarly, the Option-Critic framework uses a finite set of discrete options, each with learnable termination conditions. Because our tasks feature a continuous latent action (e.g., specifying a target joint position or velocity) and because we run on-policy PPO for the high-level layer, we found it non-trivial to combine discrete option-termination logic with continuous-time feedback.\n> Have you tried the techniques that enable stable training of other hierarchical RL algorithms such as option-critic and HIRO to stabilize the training of EvoControl instead of the annealing?\n\nThank you for the insightful question. The techniques used to stabilize other hierarchical RL algorithms do not directly apply to EvoControl as:\n* HIRO is an off-policy method that relies on importance sampling and sub-goal relabeling of previously collected trajectories. By contrast, EvoControl’s high-level policy uses on-policy PPO, which discards old data once the policy updates. Hence, off-policy relabeling and importance sampling are not feasible in EvoControl.\n* Option-critic typically defines a finite set of discrete options, each with its own policy and termination condition. Its stability arises from learning when to switch or terminate these options. In EvoControl, the lower-level controller is a single, continuous neural policy (e.g., torque commands at 500 Hz)—there are no discrete “skills” to terminate, and the entire low-level training is through ES on raw episodic return.\n\nConsequently, we rely on annealing from a PD controller to the learned neural policy, which stabilizes our on-policy training and avoids the need for subgoal relabeling or discrete option terminations.\n> For example, different learning rates for the two levels to induce almost fixed behavior of one level when the other one is adjusting to its changes. I wonder why RL has failed to train the low-level policy here (Appendix J2) as opposed to those methods.\n\nThat is a nice idea. We empirically tested a similar concept, where, when using RL to train the low-level policy, we take interleaved updates every so many steps, and also did the same with EvoControl. Interestingly taking frequent interleaved updates causes instability, and taking interleaved updates slightly less frequently improves learning (e.g., take an update of the higher-level policy for every 16 low-level policy updates), this could arise due to an update on one level has a larger effect on the overall policy than an update on the other layer.\n\n---\n*We hope that all of the reviewers’ concerns have been addressed. We’d be happy to engage in further discussions.*", "claims_and_evidence": "All claims and information are very well supported by either relevant prior work or independently demonstrated ablation experiments. For instance,​ the challenges of policy gradient that are mentioned to motivate the evolutionary algorithm in Section 3.3 are verified empirically.", "methods_and_evaluation_criteria": "The methods of the paper are in the ​vicinity of the ideas in the field and highly appropriate. The evaluations use both the standard benchmarks and customized tasks that further highlight the advantages of the algorithm.", "theoretical_claims": "The paper does not have much theory. The theoretical result seems correct from the proof sketch. I have not checked​ the details of the proof.", "experimental_designs_or_analyses": "The experiments and their analysis are mostly standard. The main concern I have is the choice to fix the number of high-level policy training steps in the comparisons instead of fixing the total environment interaction time (Line 295right). This may be specifically disadvantageous to direct torque high frequency. For this algorithm, the high-level policy takes orders of magnitudes less time to interact with the environment as each interaction is very short. I think fixing the total time and not the number of steps is more fair. If possible, including the results for this will be beneficial.", "relation_to_broader_scientific_literature": "I am aware that there are multi-level hierarchical RL algorithms for control tasks. This paper's method is highly related to those approaches. The main new ability of the new approach is that the low-level policy is not limited to a fixed behavior as opposed to the previous methods where the low-level policy was trained to reach a goal state. This gives the agent a higher representation ability for more complex tasks.", "essential_references_not_discussed": "I am not aware of any critical​ references that are missed.", "other_strengths_and_weaknesses": "The paper does a tremendous job in supporting the findings with ablation experiments and evaluations. The paper is well written and the idea is novel and interesting. Please see questions and the comment for experimental design for potential weaknesses.", "other_comments_or_suggestions": "Some typos:\n\n1. line 093 right, the definition of R, subscripts are wrongly formatted.", "questions_for_authors": "1. Are there any experiments to show the proposed method has advantages over hierarchical RL methods with goal-reaching low-level policies, such as HIRO. I saw the experiments comparing with having fi​xed PD controllers as the low-level policy, but no comparisons to HIRO. Some comments or empirical results would be appreciated.\n\n2. Have you tried the techniques that enable stable training of other hierarchical RL algorithms such as option-critic and HIRO to stabilize the training of EvoControl instead of the annealing? For example, different learning rates for the two levels to induce almost fixed behavior of one level when the other one is adjusting to its changes. I wonder why RL has failed to train the low-level policy here (Appendix J2) as opposed to those methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F5ylOLRhRR", "reviewer": "Reviewer_c2Dp", "summary": "The manuscript presents a novel bi-level optimization reinforcement learning method for high frequency control. The method combines a high-level low frequency policy with a low-level high frequency controller. Both controllers are learnable. The authors motivate the bi-level learning/optimization scheme. Overall, the manuscript, the experimental validation and result analysis are high quality. The results are also very encouraging. Finally, the authors provide motivation (and analytical proof; although trivial imho) for why a higher frequency controller is always more desirable if it can be learned effectively.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. Below, we address each concern:\n> authors do not really motivate why such MDPs exist, and why they are important\n\n### (A) Why Such MDPs Exist\n\nOur Proposition 2.1 is meant to show that there are environments/continuous-time MDPs in which acting at higher frequency can strictly yield higher returns. While this might be theoretically straightforward, it underscores the practical reality that in certain tasks, reacting faster confers a distinct advantage. Indeed, when new or unmodeled information arrives between coarser time steps, lower-frequency controllers (or those that do not adapt quickly) risk missing events that incur large penalties or lose large rewards.\n\nConcretely, we focus on safety-critical tasks where immediate reaction to collisions or unexpected contacts is crucial. In these tasks, an undetected collision or a delayed response can rapidly cause catastrophic damage. The “Safety Critical Reacher” environment (Section 6.2 of our paper) exemplifies this scenario: when there is a sudden collision, the policy must quickly backtrack, or else it will suffer a considerable penalty for the extra collision force. A lower-frequency controller often cannot respond fast enough within that sub-interval.\n\nTo further verify the existence of such MDPs, we introduce a **new “Safety Critical Halfcheetah” environment** with six degrees of freedom and an observation dimension of 19, exhibiting significantly more complex dynamics than our previous 1D safety task. Specifically, we add a blocking wall in 25% of the episodes, with collision force only observable upon contact. This wall blocks the cheetah’s forward path and incurs a penalty for any impact. As a result, the policy must quickly retreat if it detects a collision; otherwise, it continues forward. This design underscores the importance of higher-frequency actions for fast responses in unmodeled safety-critical situations, following a similar setup to \"Safety Critical Reacher\". Our full results (at https://imgur.com/a/LNBypMy) show that *EvoControl* still outperforms the baselines in this more challenging safety-critical scenario.\n\n**Table 33, Safety Critical Halfcheetah Results**\n|Same PPO high-level alg. ($\\rho$) with|Safety Critical Halfcheetah ($\\mathcal{R}$) ↑)|\n|---------------------------------------------|----------------------------------------------|\n|*Fixed Cont.* – PD Position|84.7±1.68|\n|*Fixed Cont.* – PD Position Delta|0±0|\n|*Fixed Cont.* – Random|0.0±0.0|\n|*Direct Torque Cont.* – High Freq. (500Hz)|100±1.42|\n|*Direct Torque Cont.* – Low Freq. (31.25Hz)|0±0|\n|**EvoControl (Full State)**|**120±13.9**|\n|**EvoControl (Residual State)**|**105±5.84**|\n|**EvoControl (Target + Proprio.)**|**117±34.9**|\n|**EvoControl (Target)**|**117±5.33**|\n\n### (B) Why These Scenarios Are Important\n\nThese environments mirror real robotic applications, such as:\n* *High-gain robotic arms near humans:* Where collisions must be minimized or mitigated the instant they occur, e.g., in collaborative assembly lines.\n* *Aerial vehicles under sudden gusts:* Hovering drones or helicopters must correct torque and thrust when wind forces change abruptly.\n* *Surgical robotics:* In certain procedures, micrometer-level slip or unintentional contact can cause tissue damage; responding at tens-of-milliseconds scale is vital.\n\nIn all these cases, the penalty for a delayed response can be high; hence, faster control loops can yield higher returns.\n\nFurthermore, we validated the real-world feasibility of EvoControl on a 7-DoF Franka Emika Panda (Appendix J.14, p. 51), initially demonstrating superior collision handling and reduced contact forces compared to a tuned PD controller. Importantly, the policies trained in MuJoCo XLA transferred directly to the robot without requiring additional task-specific tuning, unlike fixed-PD controllers. These results underscore the practicality of using a learned fast controller in real-world scenarios.\n> proof/claim here seems superficial ... nothing wrong with keeping it.\n\nWhile the proof (Proposition 2.1) is simple, we keep it to clarify that high-frequency control is not just an intuitive guess but can be rigorously shown beneficial under specific conditions.\n\n> not very novel (I would say the main novelty is incremental)\nThough high-/low-frequency control is common, our approach:\n1. Learns a neural fast controller (vs. fixed PD/sub-goal tracking only policies).\n2. Combines on-policy PPO for the high-level and ES for the low-level, learning jointly, avoiding instability from direct high-frequency RL.\n3. Demonstrates consistent real-world performance with minimal tuning.\n\nThis synergy extends hierarchical RL to truly fast torque control while retaining exploration benefits from slower layers.\n\n---\n\n*We hope that all of the reviewers’ concerns have been addressed. We’d be happy to engage in further discussions.*", "claims_and_evidence": "The authors back up their claims using both theoretical (although a bit weak) and empirical (quite strong) analysis.", "methods_and_evaluation_criteria": "The evaluation pipeline makes sense and the criteria is rather rigorous. I appreciate the 128 rollouts for the metrics per policy (and 3 full training runs). Overall, every question that I had while reading the manuscript was answered later in the text.", "theoretical_claims": "This might be the weakest point of the manuscript. Although I do not see anything wrong with the proof, it seems rather trivial for the specific case of MDPs that the authors assume. Also, the authors do not really motivate why such MDPs exist, and why they are important. Overall, the proof/claim here seems superficial. I would, personally, be happy with the paper without the claim/proof. Although, nothing wrong with keeping it.", "experimental_designs_or_analyses": "As I said before, the authors have really thought out the experimental analysis. The authors have included all important baselines that I could think of, and the experimental analysis is thorough.", "relation_to_broader_scientific_literature": "Continuous control lies in the heart of many real-world systems, and having more accurate, robust and highly adaptive policies can be of great importance towards the widespread adoption of robots, and RL agents in general.", "essential_references_not_discussed": "I do not think that the authors missed any important reference.", "other_strengths_and_weaknesses": "The proposed method is not very novel (I would say the main novelty is incremental), but the presentation, experimental results and analysis are of top quality.", "other_comments_or_suggestions": "I have no other comments.", "questions_for_authors": "I honesty do not have any question left. Every question that arised while reading the manuscript, the authors replied to it later in the text.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aor3pwTY8S", "reviewer": "Reviewer_J6Rg", "summary": "The paper learns high-frequency control with a two-level structure. A high-level policy working at a lower frequency is trained with PPO. A low-level high-frequency policy is obtained with evolutionary algorithms. The two-level design works better than directly training a single high-frequency control policy with RL. A learned low-level policy achieves better results than using a fixed PD controller and is robust to PD parameters. Notably, the proposed method is better in tasks that require adaptive reactions to the environment.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. Below, we address each concern:\n\n> not clear whether it directly suits control applications. Specifically, what is the inference speed of the low-level policy? Can it be deployed at its desired frequency in the real world?\n\nThank you for highlighting this point. We confirm that EvoControl can indeed be deployed at its desired frequency in real-world settings. In **Appendix J.14 (p. 51)**, we provide a **real-robot validation** using a 7-DoF Franka Emika Panda on a tabletop manipulation task, where the high-level policy operated at 10Hz and the low-level policy at 200Hz in a **zero-shot sim-to-real transfer**, https://imgur.com/a/Eq2Dewy, https://imgur.com/a/9yUONbW.\n\nWe measured the low-level network’s inference time to be **64 microseconds ($\\mu$s)** on average (over 1,000 samples), corresponding to a potential 15 kHz operating frequency—well beyond typical robotic control rates. Our real-world experiments yielded three main findings:\n1. *Stable and Fast:* The low-level network infers rapidly (64 µs per step), comfortably enabling deployment at 200 Hz or higher. In practice, the main bottleneck would typically be observation latency (on the order of a few milliseconds). Thus, the low-level controller itself is not the limiting factor for overall reactivity.\n2. *Better Collision Handling:* Compared to a tuned PD controller, EvoControl produced *lower contact forces*, highlighting the benefits of learned high-frequency torque control for safety.\n3. *No Fine-Tuning Required:* The policies trained in *MuJoCo (XLA)* transferred directly to the real robot without additional adjustments.\n\nThese results strongly support EvoControl’s practicality in real-world environments, clearly demonstrating that the high-frequency policy can indeed be deployed at the desired rate despite practical observation latency constraints.\n\n\n> For Table 3, it is better to report the raw returns without normalization, so that readers can better compare the results with other methods on the benchmark.\n\nThank you; that is a great suggestion. We already include the raw returns without normalization for Table 3 in Appendix J.7, Table 25.\n\nAllow us to kindly clarify that **all our benchmark environments** are non-standard because we increase the control frequency to 500Hz (episode length = 1000 steps, i.e., 2 seconds of real time) and remove the typical control-cost term (Appendix E). Each original Gym MuJoCo environment is typically controlled at 12.5–100Hz. Therefore, our modified settings differ substantially from standard benchmarks. We made these changes to research within robotics, the question of *whether* we can remove the lower-level fixed controllers and instead directly control the motor torques at high frequency. What benefits (e.g., faster reactions, finer control) this higher-frequency control can enable, while revealing the challenges standard learning algorithms face in these now longer-horizon tasks. EvoControl addresses precisely these high-frequency demands.\n\n>  I think experiments on similar safety-critical tasks with more complex dynamics would be more valuable for practical use.\n\nThank you for the helpful suggestion. In response, we introduce a **new “Safety Critical Halfcheetah” environment** with six degrees of freedom and an observation dimension of 19, exhibiting significantly more complex dynamics than our previous 1D safety task. Specifically, we add a blocking wall in 25% of the episodes, with collision force only observable upon contact. This wall blocks the cheetah’s forward path and incurs a penalty for any impact. As a result, the policy must quickly retreat if it detects a collision; otherwise, it continues forward. This design underscores the importance of higher-frequency actions for fast responses in unmodeled safety-critical situations, following a similar setup to \"Safety Critical Reacher\". Our full results (at https://imgur.com/a/LNBypMy) show that *EvoControl* still outperforms the baselines in this more challenging safety-critical scenario.\n\n> In line 83, CTMDP occurs without explanation.\n\nThank you for this typo; line 83 now reads \"continuous-time Markov decision processes (CTMDPs)\".\n\n---\n\n*We hope that all of the reviewers’ concerns have been addressed and, if so, they would consider updating their score. If any issues remain, we would be glad to discuss them further, especially in light of your current evaluation.*", "claims_and_evidence": "The claim that using two-level control policies can both handle long horizons and achieve fine-grained control performance is reasonable and is supported by clear evidence.", "methods_and_evaluation_criteria": "The proposed method generally makes sense, but it is not clear whether it directly suits control applications. Specifically, what is the inference speed of the low-level policy? Can it be deployed at its desired frequency in the real world?", "theoretical_claims": "I did not check the detailed proofs, but the claim seems correct from the proof sketch.", "experimental_designs_or_analyses": "1. For Table 3, it is better to report the raw returns without normalization, so that readers can better compare the results with other methods on the benchmark.\n2. The experiments on the safety critical reacher task are appreciated since the task requires high-frequency close-loop control to get good performance. However, the task has only one degree of freedom to control. I think experiments on similar safety-critical tasks with more complex dynamics would be more valuable for practical use.", "relation_to_broader_scientific_literature": "The key contributions of the paper are mostly related to RL techniques for long horizon problems, such as hierarchical RL.", "essential_references_not_discussed": "I did not notice any missing essential related works.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "In line 83, CTMDP occurs without explanation.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L5QMobMqAI", "reviewer": "Reviewer_qwWU", "summary": "The authors propose EvoControl, a hierarchical, bi-level policy network for motor control. EvoControl separates low-level high-frequency control and high-level, low-frequency control, by training two separate policies which output actions at different frequencies. The high-level policy is trained with RL, facilitating exploration and credit assignment due to the stronger impact of action noise and the more straightforward correlation between actions and their outcomes (compared to training a single high-frequency policy). The low-level policy is instead trained with evolution strategies, an algorithm less susceptible to the credit assignment problem. Experiments show that EvoControl can achieve better reward in continuous control tasks, which benefit from higher frequency decision making. This feature is especially relevant where safety is a concern.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. Below, we address each concern:\n> do the environments considered for the evaluation have the characteristics requested by the constructive proof of prop. 2.1?\n\nThank you; **Prop. 2.1** formally motivates that certain CTMDPs can yield higher returns by acting at higher frequencies. To concretely illustrate this, we use two safety-critical environments:\n1. **Safety Critical Reacher (Sec. 5.2, Table 4; App. B.1)**–Here, unmodeled collision detection requires rapid reaction to avoid a penalty. Policies with slower control frequencies respond too late and accumulate significant penalties, leading to substantially lower returns.\n2. **Safety Critical Halfcheetah** – We created a *new environment* that likewise penalizes collisions that need immediate corrective actions. We observe that policies acting faster achieve higher returns, consistent with Prop. 2.1– https://imgur.com/a/LNBypMy.\n\n> The authors should provide better context, because it is counterintuitive that locomotion, pushing or balancing would benefit so much from more frequent control....results from standard algorithms should be used to put these results into perspective\n\n**All our benchmark environments are non-standard** because we increase the control frequency to 500Hz (episode length = 1000 steps, i.e., 2 seconds of real time) and remove the typical control-cost term. Each original Gym MuJoCo environment is typically controlled at 12.5–100Hz. Therefore, our modified settings differ substantially from standard benchmarks. We made these changes to research within robotics, the question of whether we can remove the lower-level fixed controllers and instead directly control the motor torques at high frequency. What benefits (e.g., faster reactions, finer control) this higher-frequency control can enable, while revealing the challenges standard learning algorithms face in these now longer-horizon tasks. EvoControl addresses precisely these high-frequency demands.\n* **Table 3 (main paper)** reports relative policy performance at 1M high-level steps.\n* We include **explicit non-normalized results** of Table 3 in *Appendix J.7, Table 25*, alongside extended training for all baselines up to 1 billion steps in App. J.6.\n\nWe now include this in Sec. 5.\n> What does EvoControl discover that makes these robots run at unprecedented speeds?\n\nIn these high-frequency environments, standard policy learners struggle with exploration over long horizons (Peng, 2017)—EvoControl matches the exploration efficiency of temporally abstract goal-reaching methods (Fig. 2) and converges faster to high-reward policies **(App J.10, Figure 18)**. Rollouts (App. J.10, Fig. 18) show EvoControl often applies maximal torques to reach goals quickly while retaining precise control. This likely arises as direct high-frequency policy learning algorithms struggle with *less efficient exploration* and *slower convergence* (**Figure 2, Sec. 5.2**), and due to this, can get stuck in suboptimal policies (L 238, Left Col.) (App. J.6 & J.4), and low-frequency policies using a *lower-level goal based policies* struggle to learn accurate yet refined high-frequency motor movement, and rely on tuned fixed low-level controllers. We now include this in Sec. 5.2.\n> results from standard algorithms should be used to put these results into perspective ... consider pretrained policies\n\nThank you; we now **include four additional standard baselines** using publicly available Brax code of Soft Actor-Critic and PPO, for both high and low frequency control. We observe that EvoControl still outperforms, https://imgur.com/a/yVFuNKk. Pretrained policies for standard MuJoCo tasks cannot be used because our tasks differ significantly (see above). Instead, we use official high-performance Brax code to ensure the baselines are well-optimized.\n>  experiments on a real robot are only presented in the supplementary material\n\nWe appreciate this point and will incorporate real-robot results directly into the main paper (using our extra page) to showcase practical feasibility of EvoControl.\n\n> justify why they do not compare to ... hierarchical RL\n\nIn hierarchical RL, the most relevant method is HIRO, an off-policy method that assigns state-based goals to a low-level policy, rewarded by sub-goal completion. This closely resembles our *“fixed-controllers”*, which also receive state-based goals (L192). However, as shown in Table 4, sub-goal methods cannot handle the high-frequency reactions needed in safety-critical tasks. HIRO can also be unstable; with the authors’ hyperparameters and our training budget, it failed to converge and performed near-random, highlighting the need for extensive tuning.\n\n> MuJoCo Brax\n\nYes, MuJoCo with GPU acceleration (MuJoCo XLA (MJX)).\n\n----\n*We hope that all of the reviewers’ concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "1) There exist CTMDPs in which higher frequency decisions lead to higher reward. This claim is clearly true, but do the environments considered for the evaluation have the characteristics requested by the constructive proof of prop. 2.1?\n\n2) The baselines are rather ablations of EvoFormer's components. Some performance gains are completely unexpected: for example, the locomotion environments considered in this paper are practically considered \"solved\" with standard RL methods. Instead, EvoControl claims, for example, 5x performance improvement in Ant vs direct torque control (also, 2x in Walker2D, 4x in Hopper). These results need to be better motivated, because standard algorithms seem to use the capabilities of these robot nearly optimally. What does EvoControl discover that makes these robots run at unprecedented speeds?", "methods_and_evaluation_criteria": "I am not confident about the correctness of the evaluations. The authors should provide better context, because it is counterintuitive that locomotion, pushing or balancing would benefit so much from more frequent control. The authors should verify that the high frequency control does not interfere with the physics simulator, generating unrealistic behavior. Also, results from standard algorithms should be used to put these results into perspective - the fact that they are only reported in normalized form does not allow to understand whether the baselines achieve good results. I suspect the baseline performance detailed in table 3 to be low compared to the state of the art in these environments.", "theoretical_claims": "I have issues with prop. 2.1. The claim is trivial, clearly there exist processes in which taking decisions more often is an advantage. Imagine considering one environment where the reward is proportional to the number of clicks of a button, and at every decision step one can press the button or lift the finger. The authors, however, define an environment with certain requirements to write the proof, which does not seem to be related to any of the environments used in the experiments. What is the point of proving the existence of environments where higher decision frequency is beneficial, if then the requirements for such environments are not identified in any test?", "experimental_designs_or_analyses": "The environments considered for evaluation (classic control environments) do not seem to be ideal to test the importance of high-frequency control, as they involve either periodic control (locomotion), balancing (pendulum) or reaching. The authors should explain why experiments on a real robot are only presented in the supplementary material.", "relation_to_broader_scientific_literature": "The algorithm presented in this work should be compared with other forms of hierarchical control, as pairing a low-leve and a high-level controller is a common idea.", "essential_references_not_discussed": "The background and literature review sections are extensive. However, the authors should justify why they do not compare EvoControl to the cited hierarchical RL methods.", "other_strengths_and_weaknesses": "The strength of the paper is combining ES with PPO to make up for the shortcomings of each algorithm. The main weaknesses of the paper are lack of comparison with other methods and lack of analysis of very surprising results.", "other_comments_or_suggestions": "No additional comments.", "questions_for_authors": "When the authors mention that they use MuJoCo Brax, what do they refer to? MuJoCo with GPU acceleration? The environments they are considering are very popular in the RL community, they should consider pretrained policies available online for comparison and verify that EvoControl really leads to an improvement in these environments. If the results stand, they should provide an analysis of the EvoControl policy.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces EvoControl, a novel bi-level policy designed for high-frequency continuous control. The method utilizes a low-frequency high-level policy, trained with PPO, to generate latent actions. Subsequently, a high-frequency low-level policy, initially a convex combination of a PD controller and a neural network trained via an evolutionary algorithm, determines the final control action. This hierarchical approach demonstrates superior performance compared to direct learning methods and fixed PD controllers. Overall, the reviewers found the method interesting, although one questioned the novelty of the proposed bi-level policy. The authors generally addressed the reviewers' comments satisfactorily during the rebuttal, and three out of four reviewers, along with myself, believe the overall approach, experiments, and writing quality make the paper acceptable for the conference. While one reviewer (qwWU) raised concerns regarding the experiments and theoretical aspects that were not fully resolved by the authors' response, the majority opinion supports acceptance.
Decision: Accept (poster)