DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasets

Authors: xiaoyu  huang
yufeng  chi
ruofeng  wang
zhongyu  li
xue bin peng
sophia  shao
borivoje  nikolic
koushil  sreenath
u c berkeley
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 23:56:43

Abstract: Offline learning at scale has led to breakthroughs in computer vision, natural language processing, and robotic manipulation domains. However, scaling up learning for legged robot locomotion, especially with multiple skills in a single policy, presents significant challenges for prior online reinforcement learning (RL) methods. To address this challenge, we propose DiffuseLoco, a novel, scalable framework that leverages diffusion models to directly learn from offline multimodal datasets with a diverse set of locomotion skills. With design choices tailored for real-time control in dynamical systems, including receding horizon control and delayed inputs, DiffuseLoco is capable of reproducing multimodality in performing various locomotion skills, zero-shot transferred to real quadruped robots and deployed on edge computes. Through extensive real-world benchmarking, DiffuseLoco exhibits better stability and velocity tracking performance compared to prior RL and non-diffusion-based behavior cloning baselines. This work opens new possibilities for scaling up learning-based legged locomotion control through the scaling of large, expressive models and diverse offline datasets.

Main paper: 
 introduction: 
 Learning from large-scale offline datasets has led to breakthroughs in a large variety of domains, such as computer vision and natural language processing, where scaling up both the size of models and datasets leads to improved performance and generalization [1, 2] . This has led to the devel-opment of powerful generative models, like diffusion models, which are able to model complex multi-modal data distributions [3, 4] and generate high-quality images and videos.In robotics, learning from diverse offline datasets has also been shown to be an effective and scalable approach for developing more versatile policies for domains such as robotic manipulation [5, 6] and autonomous driving [7, 8, 9] . However, these domains typically involve agents that have lowdimensional action spaces (e.g., end-effector trajectory) with low re-planning frequency on inherently stable systems (e.g., robot arms or cars). For dynamical systems featuring higher degrees of freedom and more complex dynamics, such as legged robots, data-driven approaches have largely been focused on online reinforcement learning (RL) techniques [10, 11, 12] . Unlike offline learning, it can be difficult to scale online RL to both large models and datasets due to the requirements of online rollouts. Most prior works involving online RL have focused on dynamic motions with a small number of skills for each policy, and scaling towards a single policy that can reproduce a diverse set of challenging locomotion skills remains an open problem.To address the challenges of offline learning from diverse data sources and learning a set of diverse skills, we present DiffuseLoco, a framework leveraging expressive diffusion models to learn multimodalities in diverse offline datasets without manual skill labeling. Once trained, our controllers execute robust locomotion skills on real-world legged robots for real-time control.The primary contributions of this work include a novel, versatile framework that learns directly from a diverse offline dataset, demonstrating the benefits of scalable offline learning for practical locomotion skills, and a state-of-the-art multi-skill controller that learns both agile bipedal walking and quadrupedal locomotion skills within a single policy and is deployable zero-shot on real-world quadrupedal robots. Extensive real-world validation shows higher stability and lower velocity tracking errors compared to baselines, with smooth skill transitioning and robustness on varying terrains. This work demonstrates the feasibility of zero-shot learning a diverse locomotion policy directly from a static dataset, opening new possibilities for large-scale learning for real-time, low-level control for complex dynamical systems. 
multi-skill reinforcement learning in locomotion: 
 Recent works in model-free RL have demonstrated promising results in agile locomotion skills for real-world legged robots [12, 13, 14, 15, 16, 17] . Prior works have shown impressive performances on agile skills like jumping, running, and sharp turning on bipedal robots [18, 19] , and walking on two feet with quadrupedal robots [20, 21, 22] , requiring high agility and robustness. However, these skills are trained with single-skill RL and do not scale with more skills.A natural idea of learning multi-skill locomotion is to train skill-specific policies, and then coordinate through high-level planning [23, 24, 25, 26] . However, due to the coordination difficulty, these methods remain unscalable to an increasing number of skills. In comparison, learning multiskill policies from scratch typically involves parameterized motions [27, 28] with limited applicable motions, and motion imitation methods through either reward shaping [29, 30, 31] or adversarial imitation learning [32, 33, 34, 35] . However, this approach still faces challenges such as the limited expressiveness of simple models in online RL frameworks in learning diverse skills.In general, learning a diverse, agile policy with online RL remains challenging. For example, while existing RL methods have combined skills like jumping and trotting [29, 35] , trotting and standing on hind legs with wheels (without walking) [36] , a combination of more diverse and agile skills such as stable bipedal walking, pacing, and hopping in a single policy has not yet been demonstrated.Compared to online learning, offline learning offers better scalability, a simpler training scheme, and an effective way to re-use data, yet prior works in learning low-level locomotion control from offline datasets remain limited. Most prior works focus on simple simulated tasks, such as Gym locomotion tasks, with behavior cloning (BC) [37] and offline RL [38] . Among them, some leverage Q-learning on offline datasets [39, 40, 41] , or supervised learning techniques [42, 43, 44, 45] . However, these tasks are oversimplified and do not adequately consider the complexities in real-world scenarios.An alternative is the use of offline data as a foundation for online learning [46, 47] . Among them, Smith et al. develops baseline policies from offline datasets to bootstrap online learning on real robots. Yet, this approach still requires online learning. Another recent work develops offline learning on humanoid locomotion with real robots [48] , with the scope limited to only one walking skill.In comparison, the efficacy of learning completely from offline datasets, especially at a larger scale than a few simple skills, remains unproven in legged locomotion control.Emerging efforts to apply diffusion models to real-world robots include using diffusion to manage a variety of manipulation tasks with visual inputs and incorporating self-supervised learning and language conditioning [49, 50, 51, 52] . Yoneda et al. leverages the reverse diffusion process for shared autonomy with a human user in end-effector planning. Additionally, hierarchical frameworks are being developed to handle tasks requiring multiple skills, pushing towards generalist policies [54, 55, 56] . However, these prior works primarily focus on high-level planning on manipulation systems, featuring a low-dimensional action space (e.g., end-effector position), low-replanning frequency (e.g., around 10 Hz), and inherently more stable dynamics.In contrast, using diffusion models for high-frequency, low-level control remains limited [57] . The most relevant work uses online RL to train diffusion-based actor policies in simpler simulation settings [58] , but transitioning to hardware with high-frequency feedback control presents significant challenges due to the instability and rapid dynamics of legged robots [59] . The key difference of this work is demonstrating the applicability of diffusion models in high-frequency, low-level legged locomotion control and the advantages of multimodality and scalability in the real world. 
versatile framework for diffusing locomotion from offline dataset: 
 In this section, we introduce DiffuseLoco, a framework for generating and utilizing offline datasets to train scalable multi-skill locomotion policies. DiffuseLoco uses diffusion models to train a lowlevel multi-skill locomotion policy from datasets with diverse agile skills. A schematic illustration is shown in Figure 2a . Our framework consists of three stages:Data Source We start by collecting or using an existing offline dataset of multiple skills. To generate the data, we first obtain single-skill control policies, called source policies, conditioned on goals g (e.g., velocity commands and base heights). The source policies can be trained using different methods, and while we assume the same control frequency, their observation and action spaces can still vary significantly. Thus, we instead collect a set of source-agnostic state-actiongoal pairs across all source policies. For legged robots, the widely-used source-agnostic states and actions are proprioceptive feedback s t from the robot and joint-level PD targets a t . We only collect successful episodes. Although we use simulation data here, DiffuseLoco efficiently re-uses data and can scale to more expensive real-world data collection. More details are in Appendix F.Training In the second step, we train our DiffuseLoco policy from the offline dataset in an endto-end manner. Let input state and goal history length be h and output action prediction length be n. During training, we sample a segment of state trajectory s traj and corresponding action and goal sequences, a traj and g traj . We sample a diffusion step k randomly from {1, . . . , K}, and sample a Gaussian noise ϵ k to add to the action sequence. Then, a transformer-based denoising model takes the noisy action sequence along with states trajectory s traj , goal trajectory g traj , and diffusion step k as input, and predicts the added noise as ϵ θ . The predicted noise ϵ θ is then regressed to match the true noise ϵ k with mean square error loss. In this way, the denoising model is learned to generate sequences of low-level actions conditioned on robot states and goals from the dataset. Details of the model architecture and training objective are introduced in Section 4.In the last stage, we zero-shot transfer the trained DiffuseLoco policy to the robot hardware. During deployment, the DiffuseLoco policy takes a sequence of pure noise sampled from a Gaussian distribution and denoises it conditioned on states s traj and the goal g traj . The denoising process is repeated for K iterations to generate a sequence of actions, but only the immediate action a t is used as the robot's joint-level PD targets. After executing this action, the policy takes a new sequence of states from the robot and updates the immediate action from the newly generated sequence. This aligns with the Receding Horizon Control (RHC) framework, instead of interpolating the action sequence at high frequency in prior works [50, 60] . RHC allows DiffuseLoco to replan quickly with fast-changing robot states to ensure up-to-date actions while considering future steps. However, due to the large number of parameters and denoising steps, we must accelerate inference to meet the RHC control frequency. The acceleration techniques are detailed in Appendix G, which enable running the DiffuseLoco policy on an edge-compute device mounted on the robot. 
diffusion model for real-time control: 
 In this section, we introduce the diffusion model backbone, shown in Figure 2b , with a special focus on design choices for real-time control and inference acceleration.DDPM for Control To model multi-modal behaviors from diverse datasets, we leverage Denoising Diffusion Probabilistic Models (DDPM) [61] with a transformer backbone. DDPM is a class of generative models where the generative process is modeled as a denoising procedure, often referred to as Stochastic Langevin Dynamics, expressed in the following equation,EQUATIONwhere N (0, σ 2 I) denotes the sampled noise, α, γ, and σ are its hyperparameters. For clarity, we now use subscripts t-a:t-b to indicate trajectories from timestep t -a to t -b. First, an initial noisy action sequence, a K t:t+n , is sampled from Gaussian noise, and the DDPM conditioned on states s t-h:t , goals g t-h:t , and previous actions a t-h-1:t-1 undergoes K iterations of denoising steps.Unlike previous works applying DDPM in manipulation [49] , the inclusion of previous actions, i.e., I/O history, is crucial to better perform system identification and state estimations for dynamic legged control, as evaluated in [18] . Furthermore, instead of concatenating state and goal into a single embedding [49, 50] , we leverage the transformer's attention mechanism to assign different attention weights to separately embedded rapidly-changing I/O history and relatively static goals, enabling the policy to adjust focus between adapting to dynamic environments and achieving goals.The denoising process yields a sequence of intermediate actions characterized by progressively decreasing noise levels: a K , a K-1 , . . . , a 0 , until the desired noise-free output, a 0 , is attained. This process can be expressed as the following equation:EQUATIONwhere a k t:t+n represents the output at the k th iteration, and ϵ θ (a k t-h-1:t+n , s t-h:t , k) represents the predicted noise from the denoising model ϵ θ , which is parameterized by θ, with respect to a k t-h-1:t+n , s t-h-1:t , and iteration k. During training, we opt to use the simplified training objective proposed by Ho et al. [61] ,l = M SE (ϵ k , ϵ θ (a t-h-1:t+n + ϵ k , s t-h:t , g t-h:t , k)) .(EQUATIONwhere ϵ k is the sampled noise at iteration k. Appendix B describes further architecture details.To achieve real-time deployment, we predict current actions using delayed inputs, overcoming the long inference times of large models like transformers that exceed the 30Hz frequency requirement.In DiffuseLoco, we use one-step delayed inputs-s t-h-1:t-1 , a t-h-2:t-2 , and g t-h-1:t-1 -to predict current actions a t:t+n . Initiating inference before receiving the current state s t allows parallel processing and ensures up-to-date actions. We adopt this design for DiffuseLoco because it learns sequences of action predictions, suitable for generating actions further in time, and handles higher input delays better than small-scale MLP policies, which typically manage delays less than one control step, for instance, 25% less than DiffuseLoco as noted in [18, Table IV ].Sampling Techniques To accelerate diffusion models during robotic deployment, prior work often uses samplers like the Denoising Diffusion Implicit Models (DDIM) [62] , which employ a deterministic process to reduce sampling steps and speed up inference, albeit with some quality loss. However, in real-time legged control, DDIM's less accurate outputs increase step-by-step errors, leading to higher compounding errors and more out-of-distribution scenarios during deployment. Thus, we adopt DDPM for its enhanced robustness and performance in real-time control. The ablation can be found in Appendix D.3. 
results: model capacity: 
 To scale up learning locomotion skills as discussed in Section 1, the critical questions are whether DiffuseLoco can (a) be trained with various sources of demonstrations and (b) incorporate a diverse set of skills present in the dataset. We answer these questions by presenting a state-of-the-art fiveskill policy that combines four quadrupedal skills, and more importantly, a bipedal locomotion skill for quadrupedal robots, not yet demonstrated by prior RL frameworks.Learning from Diverse Data Sources Figure 3 illustrates the five skills DiffuseLoco acquires for a quadrupedal robot: trotting and pacing (trained with AMP [32] ), hopping and bouncing (trained with nominal CPG curves [27] ), and agile bipedal walking (trained with symmetry augmentation [63] ). After collecting demonstrations of these skills separately in simulation, we directly learn from this combined dataset and achieve robust zero-shot transfer to actual hardware. This capability surpasses previous offline learning methods that were largely confined to simulated environments with simplified dynamics, showcasing DiffuseLoco's robust real-world performance.The ability to learn from diverse skill sources is crucial for scaling locomotion learning. For instance, bipedal walking requires specific early termination conditions and reward landscapes, unlike quadrupedal skills. Also, while basic symmetry augmentation yields effective motions, prior methods like AMP [12, 36] require complex trajectory optimization as reference, posing significant challenges. Despite different requirements for observation, action spaces, and auxiliary signals across RL methods, DiffuseLoco is able to accomplish all skills with only basic proprioceptive inputs. Skill Transitioning We emphasize DiffuseLoco's capacity to transition freely between skills without transition data in the dataset, such as from hopping to bipedal walking and then to pacing, as shown in Figure 1 . This sequence highlights DiffuseLoco's robustness against variations in starting state and the stability required to execute these skills successfully. Additional skill transitions are available in Appendix A. We also analyze quantitatively on transition smoothness in Appendix C.In addition to transitions under different goals (commands), DiffuseLoco also demonstrates both trotting and pacing under the same goal. As shown in Figure 4 , the policy begins with trotting and only switches to pacing when a sudden braking event significantly alters the contact sequence. This highlights DiffuseLoco's effectiveness in learning and adhering to different modes from the offline dataset, committing to a single mode within each rollout unless prompted by external disturbances.Extension to Bipedal Robots In addition to quadrupedal robots, we also demonstrate the effectiveness of our method on high-dimensional, highly non-linear human-sized bipedal robot in the MuJoCo simulation [64] . First, we collect demonstrations evenly from two separately trained single-skill RL policies on walking and running, adapted from [18] . After training directly on this aggregated dataset, our method successfully learns both skills within a single policy. Furthermore, as shown in Figure 1 , our policy can transition from walking to running smoothly without specific transition data in the dataset, in addition to maintaining each skill's stability before and after transitions. This demonstrates one of the initial working combinations of these skills on bipedal robots.Robustness To demonstrate DiffuseLoco's robustness, we show both quadrupedal and bipedal locomotion over different ground conditions, including padded floor, bare floor, turf, and over small terrain variations, in Figure 5 . Notably, bipedal walking over a half-padded floor shows high robustness to differences in ground height, friction, and restitution forces on each of the standing legs. Compared to DAgger, DiffuseLoco captures the randomized dynamics via offline data only.In conclusion, we demonstrate that DiffuseLoco can learn a diverse set of skills from various offline sources and specialized RL frameworks. Importantly, DiffuseLoco shows better scalability in learning diverse and agile locomotion skills that existing RL frameworks have not yet illustrated. It also demonstrates robustness, skill transitions, and the ability to extend to more complex legged robots. 6 Results: Quantitative AnalysisIn this section, we first compare DiffuseLoco against multi-skill RL (AMP) and non-diffusion BC baselines. Since there is no RL baseline with all five skills, we focus on pacing and trotting in this analysis. Next, we ablate on the key model and dataset characteristics, such as model size and skill size to study their impact on performance. Detailed task setups are provided in Appendix E.We first compare DiffuseLoco with RL-based multi-skill control policy AMP. Table 1 shows that DiffuseLoco is the only method among the baselines that is able to reliably complete all trials without falling over. Specifically, the RL-trained AMP and AMP with history inputs (AMP w/ H) baselines struggle with low and high speed commands. For 0.3 m/s forward command, the actual velocity is more than 90% slower than the commanded velocity. For 0.7 m/s forward command, they achieve a stability metric of 0% and 20% respectively.This shows the prevailing problem of mode-collapse for Generative Adversarial Networks (GAN), where the generator becomes overfitted to a limited range of outputs that are often similar or identical [65, 66, 67] . In AMP, the actor overfits the simulation environment, losing generalization to new environments (e.g. real world), but reverting to early stage behaviors where the discriminator is not yet converged. An example is in low-speed tasks where it oscillates in place with correct frequency but reduced amplitude, similar to its behaviors in early training stage.In contrast, DiffuseLoco generalizes within the expert demonstrations and does not revert to infeasible actions. Moreover, DiffuseLoco with its diffusion model efficiently learns the multi-modality of the different skills for the same command, enabling it to perform valid locomotion skills without mode-collapsing. In general, DiffuseLoco achieve both better stability and velocity tracking performance compared with AMP-based baselines.We further compare DiffuseLoco with non-diffusion BC methods, such as transformer with one-step prediction (TF) or with a prediction horizon (TF w/ RHC). Shown in Table 1 , DiffuseLoco outperforms TF and TF w/ RHC in both stability and robustness. Specifically, TF lacks robustness and fails the 0.7 m/s forward and left turn tasks completely, likely because single-step action prediction makes the policy less aware of future actions, leading to more jittering behavior.With receding horizon control, TF w/ RHC overcomes most of the jittering problem and can complete most of the tasks. However, for more agile motion such as 0.7 m/s forward, the stability metric drops drastically to merely 40%, likely due to the reconstruction loss used in TF w/ RHC training tends to overfit the action trajectories in the dataset, resulting in less robust policy in the out-of-distribution scenarios (e.g. real world).In comparison, our DiffuseLoco shows more stable and smooth motions measured by both stability metrics and magnitudes of the body's angular velocity. On average, DiffuseLoco achieves 10.40% less in magnitude for the body's oscillation over all trials. As a result, the smoother locomotion skill leads to on average 38.97% less tracking error compared to TF w/ RHC. Thus, we believe diffusionstyle training is more suitable for locomotion tasks than previous Behavior Cloning methods. Ablations We perform ablations to study the key characteristics. We report the average live time and the percentage of episode timeouts (success) on bipedal walking across 100 randomized environments with 20-second maximum length. As shown in Table 2 , predicting only the next action leads to jittery movements and worse performance, while predicting too far into the future (16 steps) also degrades performance due to higher variance in future actions. The optimal performance is achieved with moderate prediction steps. For model size, performance consistently improves with larger parameter counts, confirming that DiffuseLoco scales with model capacity, and the large expressive model is necessary to handle the challenging five skills. For number of skills, we compare the multi-skill policy with a single-skill policy (bipedal walking) trained on the same shard of bipedal data. We find that multi-skill training slightly improved performance, likely because it exposes the policy to a broader range of dynamics, enhancing generalization and robustness. Thus, DiffuseLoco scales with increasing number of skills. Further ablations on design choices are in Appendix D. 
limitation: 
 Admittedly, DiffuseLoco's robustness is subpar compared to the state-of-the-art single-skill RL. For instance, the sim-to-real gap is larger on bipedal hardware due to more complex dynamics requiring more accurate data. Fortunately, DiffuseLoco can learn directly from offline real-world data without accessing expert policies or repeated data collection. Appendix D.5 shows the ablation of improving the robustness by absorbing data from various dynamics. By expanding our dataset with more diverse, potentially real-world data, we anticipate its robustness will improve progressively. In this section we present more results on a series of video clips to showcase skill transitioning of DiffuseLoco. Here, we explain the details of the receding horizon control framework and diffusion model's architecture, dataflow, and transformer backboone.Learning sequences of actions instead of single-step action in training helps in improving temporal consistency of the policy. However, in dynamic systems such as legged robots, the error accumulates significantly after a short horizon of planned steps, and the predicted actions further ahead may no longer be useful for control. Therefore, we adopt the Receding Horizon Control (RHC) manner, where DiffuseLoco policy generates n steps of actions, but only executes the very first step of actions. This is in contrast to previous work which infers a sequence of actions at a lower frequency and uses interpolation to get a high-frequency action [60, 50] . Such a setup allows us to replan rapidly with fast-changing states of the robot while keeping future steps in account. As we evaluated in Appendix D.2, using RHC is critical in improving smoothness and consistency of a legged locomotion control policy.The DiffuseLoco policy leverages an encoder-decoder transformer DDPM. First, the past robot's past I/O trajectory (s t-h-1:t-1 , a t-h-2:t-2 ) and given goal sequence g t-h-1:t-1 are transformed into separate I/O embedding and goal embedding by two 2-layer MLP encoders, respectively. Then, we sample noise ϵ(k) for diffusion time step k with the DDPM scheduler and add to the ground truth action a from the offline dataset to produce a noisy action a k t:t+n = a t:t+n + ϵ k . The noisy action a k t:t+n is then passed through an MLP layer into action embedding. The noisy action tokens are then passed through 6 Transformer decoder layers, each of which is composed of an 8-head crossattention layer. Each layer computes the attention weights for the noisy action tokens querying all the state embedding, goal embedding, and the timestep embedding reflecting the current diffusion timestep k. We apply causal attention masks to each of the state embeddings and goal embeddings separately. The predicted noise ϵ θ (a t-h-2:t+n , s t-h-1:t-1 , g t-h-1:t-1 , k) is then computed by each corresponding output token of the decoder stack. We then supervise the output to predict the added noise with Eqn. 3 to find optimal parameters θ of the denoising model ϵ θ .Using delayed inputs is uncommon in RL-based locomotion control but is necessary here due to the longer inference time of diffusion models compared to MLPs. Unlike prior works (mostly based on RL) which commonly consider the zero-order-hold communication delay and ignore the inference delay, DiffuseLoco's inference takes 10-20ms on different GPUs (50-100% of a control cycle), making the latter much more significant. To demonstrate the impact of this delay, we run an ablation without delayed inputs and test with inference delay. With 10ms inference (on the most recent RTX 4090), the robot continued walking with minor staggering, but at 15ms, it cannot stand up anymore.The hyperparameters are summarized in Table 3 As shown in Figure 4 , the mean and standard deviation (STD) of base angular velocity during skill transitions are similar to steady single-skill execution, indicating comparable body oscillation. The maximum value during transitions does not exceed that of single-skill execution, suggesting stable body dynamics. These statistics come from a single trial, with STD reflecting angular velocity fluctuations over time. A higher STD indicates more fluctuations, which is undesirable. As some fluctuation is expected during locomotion, an STD around 0.4 is normal. Transitions, like from hopping to trotting, may involve acceleration and deceleration, so the mean value may not simply be an interpolation. We define smooth transitions as having minimal abrupt oscillations and consistent angular velocity, though interpretations may vary.In Figure 13 , we present the joint position trajectories during the skill transition from hopping to pacing. The plot illustrates that after receiving the new command (indicated by the vertical dashed line), the policy does not switch to the pacing mode immediately. Instead, it completes the current gait cycle before transitioning. During the subsequent cycle, the system interpolates from the current joint position to a suitable starting position for the new skill. Following this interpolation, the system initiates the pacing gait and gradually adjusts the joint movement magnitudes from the interpolated state towards steady-state execution. We note that naively switching between different skill policies would lead to abrupt changes, as the system would immediately begin executing the new skill without proper interpolation.In Figure 14 , we show a plot of the command velocity v.s. actual velocity on bipedal robot Cassie in Mujoco simulation involving a transition from walking to running. Although some steady-state error exists in low-speed walking, we see that the robot is able to achieve and maintain a steady running speed of 2.0m/s as commanded.Here, we further evaluate the design choices used to build DiffuseLoco policy in simulation and the real world by extensive ablation studies. We use the same experiment setups as in Section 6, detailed in Appendix E. Table 5 : Performance Ablation Study across different ablations and DiffuseLoco policy in realworld experiments. Stability (the higher the better) measures the number of trials in which the robot stays stable and does not fall over. E v (the lower the better) measures the deviation from the desired velocity in percentage. The experiments are conducted with different command settings (Left). Each command is repeated non-stop for five trials, and we report the average and standard deviation of the metrics across five trials.To validate our design choices, we ablate DiffuseLoco with the following critical components and compare them to our real-world benchmark.• Without Receding Horizon Control (DL w/o RHC): Replace RHC with one-step prediction in an autoregressive manner and keep the diffusion model. • U-Net as the backbone (U-Net): Replace the Transformer with a U-Net as the backbone, adjusted to the same parameter count.To isolate the effects of RHC, we test a variant of DiffuseLoco without RHC (DL w/o RHC), finding that it struggles with faster speed goals and exhibits significant jittering behaviors, as detailed in Table 5 . This suggests that single-step token-prediction models like GPT are less suitable for legged locomotion control than diffusion models, which predict sequences of future actions.As discussed earlier, popular diffusion-based frameworks like DDIM often reduce sample iterations for inference acceleration, trading off output quality for speed, often with ten times fewer iterations [62] . While this approach suits tasks like image generation, which tolerate some variance, it underperforms in quadrupedal locomotion control. As shown in Table 5 , both DDIM-100/10 and DDIM-10/5 exhibit worse stability and higher velocity tracking errors. Noticeably, the 100 training steps and 10 inference steps variant demonstrates limping behavior and fails two trials. Tracking errors for both variants increase by 50.69% and 42.04%, respectively, compared to DiffuseLoco.Thus, we believe that noisier control signals from the DDIM pipeline likely disrupt the control of inherently unstable floating-based dynamic systems, like legged robots. An interesting future work direction could be on control-specific sampling techniques to accelerate diffusion models without compromising stability and performance.In addition, we compare against another commonly used architecture in diffusion models, a CNNbased U-Net as the backbone of DiffuseLoco. Qualitatively, the U-Net policy is shaky and inconsistent, and quantitatively, it has one of the highest errors in velocity tracking, with worsened stability due to its shaky actions. We reckon that this is because CNNs are not the best fit for temporal data and also lack separate attention weights for goal conditioning. This is consistent with prior work [49] that finds U-Net underperforms Transformer, especially in high action-rate dynamical systems.Lastly, we explore how dataset characteristics influence the robustness and performance of Diffuse-Loco in real-world scenarios. Consistent with previous findings that diversity in training data, such as noise insertion, mitigates compounding error [68] , we demonstrate that increasing the variety of dynamics parameters in simulation environments where we collect data also enhances robustness.As in Table 5 , training DiffuseLoco on a dataset with dynamics randomization leads to a 44.26% increase in both robustness and stability compared to DL w/o Rand baseline. Specifically, in the challenging 0. Here, we evaluate the impact of goal-conditioning, hypothesizing enhancements tracking performance and stability. In the DL w/o Goal baseline, we do not add the goal-conditioning encoder. Instead, the goal is concatenated with the robot's I/O. Considering the noisy base velocity estimations on real robots, we utilize simulation environments with extensive dynamics randomization to perform a large number of trials and get more systematic results. As shown in Figure 15 , Diffuse-Loco achieves a 15.4% reduction in linear velocity tracking error and a 14.5% reduction in angular velocity error compared to the DL w/o Goal baseline. Moreover, over 64 trials with identical commands, DL w/o Goal falls over four times, or 6.25% of all trials, whereas DiffuseLoco experiences no failures. This pattern persists in real-world testing, where DL w/o Goal fails one trial in a 0.7m/s forward test.These results underscore the importance of goal-conditioning with distinct attention weights for dynamic system control, revealing that the robot's I/O history and goals, governed by physics and arbitrary objectives respectively, should not be merged into one embedding space.We find that a pre-trained policy indeed provides a solid starting point to finetune other skills with efficient samples. For example, after training on 500 episodes of only quadrupedal trotting skill, we finetuned this policy on only 10 episodes, 10,000 steps (200-second long) in total, of hopping data and found that 10 episodes were enough for the policy to learn this new skill. In terms of using real-world data to finetune with the flat-terrain policy, we expect we can collect a small number of trajectories collected in the real world to finetune the policy. We mark finetuning with real-world data as an interesting and feasible future work.This analysis includes walking for four meters under five goals (commands) with different velocities. The goals are the following: move forward at three different speeds: 0.3 m/s, 0.5 m/s, and 0.7 m/s, and make a left turn and a right turn at 0.3 rad/s. We record the actual linear velocities via a Kalman filter state estimation [69] , and the number of trials where the robot does not fall over through out the trial as the stability metrics. We repeat each experiment five times consecutively and report the mean and standard deviation across five runs.Skill information are often unscalable or unavailable during training and deployment. For a boarder range of applicability, we limit the scope of comparisons within not-skill-conditioned multi-skill RL and non-diffusion BC baselines. Specifically, the RL baselines include,• Adversarial Motion Priors (AMP) [32] : An MLP policy trained using AMP with RL (PPO) and style reward of both pacing and trotting reference motions. We directly use the opensourced checkpoint from [32] . We note that although several skill-conditioned RL policies [34, 35, 33] have been introduced since [32] , yielding better sim-to-real results, progress in unconditioned multi-skill policies has been limited.• AMP with history steps (AMP w/ H): To align with DiffuseLoco, we train an AMP policy with 8 steps of state and action history with the same setup as [32] and a similar evaluation return in simulation.Furthermore, we compare DiffuseLoco with non-diffusion BC policies, which can be categorized into autoregressive token prediction [43, 70] and action sequence prediction as used in [71] . We adopt baselines for each category.• Transformer with Autoregressive Token Prediction (TF): A Generative Pretrained Transformer (GPT) [72] policy similar to a decision transformer [43] without reward conditioning. This only generates one timestep action.• Transformer with Receding Horizon Control (TF w/ RHC): A transformer policy with the same future step action predictions. The model's architecture is identical to our Diffuse-Loco model, but it directly predicts future action sequences and the loss is replaced by the reconstruction loss l = M SE(π θ (s t , g t ), a t ).These baselines have the same parameter count of 6.8M and are trained with the same learning rate scheme and number of epochs as DiffuseLoco.Remark 1 Typically, previous work uses DAgger style algorithms [73] to better cope with distribution shift, but these methods require access to the expert policy in training and online learning environments. As a more scalable and versatile framework, we limit our focus to learning entirely from offline datasets.After the architecture details, we introduce the details in creating the offline locomotion dataset used in this work. We will explain the state, goal, and action spaces, followed by a brief introduction to the source policies and dynamics randomization used to diversify the data. We collect a total of 4 million data of state-action-goal pairs in the offline dateset for the quadrupedal robot tasks, and 10 million transitions for the bipedal robot tasks.The state space is the robot's proprioceptive feedback. In the quadrupedal locomotion control case, this consists of the measured motor positions q m , measured motor velocities qm , base orientation q ψ,θ,ϕ , and base angular velocities qψ,θ,ϕ . Note that we exclude quantities from the estimation of base velocity ( qx,y ) to prevent additional estimation errors.The goal of the locomotion task is the commands given to the policy. For quadrupedal robots, the command includes desired sagittal velocity q d x in the range of 0 m/s to 1 m/s, desired base height from 0.2 m to 0.6 m, and desired turning velocity q d ψ from -1 rad/s to 1 rad/s.The action space is the robot's joint-level commands. In this work, we use the desired motor position q d m as the action. This is then used by joint-level PD controllers to compute motor torques τ at a higher frequency.We obtain source policies using three different RL methods. We leverage proximal policy optimization (PPO) [74] to optimize each of the source policies, and we train the policies in simulation (Isaac Gym [14] ). We evenly distribute the data generated from each of the source skill-specific policies.For skills trained with AMP, we provide a reference motion retargeted from motion capture data of a dog [12] , and incorporate an GAN-style discriminator to encourage the robot to imitate the reference motion without extended reward engineering. Then, the reward of this method is formulated as motion imitating term (provided by the discriminator [75] ) and task term (e.g., tracking error, etc). For fairness, we would like to highlight that the AMP baseline used in quantitative comparison in Section 6 and the data-generating AMP policies achieve the same level of performance in simulation.For skills trained with CPG-guidance, we follow the formulation in [76] and provide nominal reference motions of strictly periodic motions generated by a Central Pattern Generator with phase signals. Specifically, for hopping skill, the phase selections for all legs are 0, and for bouncing skill, the phase selections are 0 for the front legs, and π for the hinder legs. The reward is composed of task term (e.g., velocity tracking error, etc), motion tracking term (e.g., reference motion tracking error, etc), and smoothing terms (e.g., action rate, etc).We train the bipedal locomotion skill for quadrupedal robots following a symmetry-augmented RL policy [63] to achieve a symmetric gait pattern that is crucial for sim-to-real transfer. Specifically, the data collection process is augmented by the addition of symmetric states and actions. The reward includes task term (e.g., velocity tracking error, etc), gait pattern term (e.g., feet clearance height, etc), and smoothing terms (e.g., action rate, etc).Although the source policies used to collect the dataset in this work are all RL-based policies, our framework is general and can include the data generated from model-based optimal controllers (such as from [77] ) and others. The requirement is to align the state and action spaces among different source policies, and the frequency of the policy should be kept the same.In order to diversify the training dataset for DiffuseLoco, we also include the same amount of dynamics randomization [78] during the training of the source policies and the data generation using these policies. Specifically, in each episode in simulation, the dynamics parameters are randomized. These include the motor's PD gains, the mass of the robot's base (up to the weight of the onboard compute), ground frictions, and random changes in base velocity. The randomization ranges are adapted from the source policy's original methods.Although the diffusion model targets real-time use, it cannot meet the real-time targets without further tuning. Compared to previous works using Transformer for locomotion control with 2M parameters [79] , our model is 3 times larger in parameter count (6.8M parameters) and 10 forward propagations are needed in each inference. Thus, an additional effort is needed to accelerate the diffusion on the edge computing device on the robot, such as the setup shown in Figure 17 . In this section, we explore several methods to accelerate the inference process of the diffusion model to enable it to run real-time onboard.Our DiffuseLoco policy has a parameter count of 6.8 million parameters, which exceeds most modern mobile processors' cache capacity. Furthermore, hardware on a typical consumer-grade central processing unit (CPU) is not optimized for the operators used in transformer networks. The graphics processing unit (GPU) is more suitable for computing the high-dimension matrix and vector operations. To ensure the portability of the setup, we use an accessible NVIDIA Mobile GPU as the deployment platform. For real-time deployment, an acceleration pipeline is built in the DiffuseLoco framework to convert and optimize our model towards the target compute platforms. The operators of the model are first extracted with ONNX [80] . Then, TensorRT is used to refine the execution graph and compile the resulting execution pipeline onto the target GPU. Through domain-specific architecture optimizations, the operations and memory access patterns are optimized to utilize the full capability of the GPU. With this approach, the speed for each denoising iteration is increased by about 7X compared to the native implementation in PyTorch, and the maximum inference (with 10 denoising iterations) frequency is increased from 17.0 Hz to 116.5 Hz. To showcase the effect of this acceleration approach, we conducted a benchmark on the inference frequency of the policy running on multiple hardware platforms we have access to, shown in Figure 16 . With the help of the acceleration framework, the compute platform can be deployed onboard a Go1 quadruped robot. A mini-computer equipped with Intel Core i7-13700H and NVIDIA GeForce RTX 4060 Mobile is attached to the top of the robot, as showcased in Figure 17 . This computer runs DiffuseLoco policy and is powered by a dedicated battery bank, separated from the robot's internal battery. This arrangement is capable of running the policy for up to 90 minutes. The minicomputer connects to the robot via an Ethernet cable to send action for the joint-level PD controls on the robot's computer. We note that all the experiments we present are completed on the same edge computing device. 
: 
 Figure 1 . Snapshots of (top) a quadrupedal robot trotting with DiffuseLoco control policy via onboard computing; (middle) DiffuseLoco policy performing a sequence of challenging skills, including hopping, bipedal locomotion, and pacing with smooth skill transitioning; (bottom) a bipedal robot with DiffuseLoco transitioning from walking to running stably. We encourage the readers to watch supplementary videos on these runs. 
 
 Appendix:  


Decision: All the reviewers agree that this paper is good, providing an interesting contribution to the literature.
In general, reviewers praise the fact that the authors can show multimodal behavior (bipedal and quadrupedal locomotion)  and that the authors make real-time inference possible for a relatively large transformer-based diffusion policy.

However, one reviewer also states that the only relevant contribution to this work is the codebase. In my opinion (and in contrast to what the reviewers find fundamental), this issue is a pretty strong point against acceptance, as the scope of the contribution for a major conference such as CoRL cannot be limited only to implementation.

I am familiar with this work, and I believe the contributions could go slightly beyond the simple implementation. However, I also believe that the authors should definitively clarify some points and improve other aspects, specifically:
- The claim of smooth transition doesn't seem to be sufficiently substantiated by the experiments. The authors should provide a more in-depth experimentation of this claim. indeed, this claim seems not to be supported by the video material
- The authors should justify the size of the model for a relatively simple task (learning in state space a limited amount of locomotion skills) as a reviewer suggests, this ablation is missing and is particularly crucial for the paper, as the contribution from the scientific point of view is relatively limited.
- Quantitative results for bipedal robot experiments are missing.
- The baseline performance seems to be quite poor. The authors should provide evidence that the baselines are implemented correctly and explain the failure modes more in detail.
- It is fundamental that the authors clarify all the questions of the reviewers, in particular regarding data collection. Looking again quickly at the paper, the authors seem to claim they are using Symmetry Augmented RL. The authors should clarify in detail this. Furthermore,  it would be advisable to add a simple baseline using directly the policies learned with Symmetry Augmented RL to prove that the diffusion model actually can fuse skills better than naively switching these policies.
- If the codebase is the major contribution, I expect this codebase to be available, at least at a preliminary stage, for the rebuttal period. The code base should be executable, and I'll try to check that it can be run following the instructions. Without the codebase, I would not recommend this paper for acceptance.

In general, this paper seems a reasonable contribution, but I would advise the reviewers to carefully check the response of the authors to their points and my comments, and possibly try if it's possible to run the author's code in simulation, as the technical part seems to be prevalent in their work.

===

The reviewers were convinced by the rebuttal and one of the reviewers also was able to test the code, and confirmed it was working.
While I had problems executing the code (due to hardware limitations) I still checked that the instructions were detailed enough and that the overall codebase was reasonable.
There are two main issues remaining: one is that is not clear how to reproduce the gait switching experiments, which should be the main contribution of this paper, and the other is that the codebase still requires a bit of cleanup.
However, given that the authors handled all the issues of the reviewer and provided an open-source link to their code, I believe that the last remaining issues can be easily fixed by the camera-ready version of the paper.

Reviews: review: title: title: A novel diffusion model-based approach for locomotion policies.
paper_summary: paper_summary: The paper presents DiffuseLoco, a framework leveraging diffusion models to train multi-skill legged locomotion policies from offline datasets. This framework aims to address the challenges associated with scaling up learning for legged robots by providing a scalable solution that can handle multiple skills within a single policy. DiffuseLoco emphasizes real-time control and robustness in dynamic environments, demonstrating successful zero-shot transfer to real quadrupedal robots.
main_review: main_review: Strength:
The paper is well-motivated and very well-written and the results are very comprehensive. 
The idea of using diffusion for policy representation has been done in manipulation but its application in real-world locomotion is not only novel but also more challenging. 
The results section is comprehensive and impressive.



Weakness:
A limitation of such a method can be the length of the trajectory generated. If the task horizons were longer, how does DiffuseLoco scale? The paper uses a relatively small predicted length of 4, an ablation on this on the mujoco humanoid can be really helpful. If this turns out to be a limitation, it should be added to the limitation section.

Considering acceleration was necessary for deployment, was an ablation performed on the size of the model used? How does a smaller model compare to the 6.8M model used in the paper? This ablation can give more insight into how such a model would scale.

Regarding robustness, can you provide a brief explanation as to why the model is robust to different ground conditions? It is not directly obvious why an imitation learning model would be robust to state distribution shifts. Is this robustness a consequence of the diffusion model or high-quality single-skill control policies that generated the data?
questions: questions: What is the effect of delayed inputs? Is this a standard implementation detail in locomotion? If yes, please provide a reference. If not, can you provide a simple metric for how much the delayed input helps? Why just a delay of one sample?

The idea of using diffusion for humanoid locomotion from multiple skills appears in [1] as well, albeit they use diffusion to generate a policy model instead of an action sequence, can you provide a brief statement on how DiffuseLoco compares to this? They also achieve strong skill transitioning. 

How does DiffuseLoco perform on new tasks/ environments, specifically how many trajectories would be needed to finetune a pre-trained model? The paper mentions that better-quality real-world data can alter performance. How many trajectories of real-world data would be needed to finetune/cotrain the DiffuseLoco model?


[1]: Hegde, Shashank, et al. "Generating behaviorally diverse policies with latent diffusion models." Advances in Neural Information Processing Systems 36 (2023): 7541-7554.
limitations: limitations: 2
review_summary: review_summary: The paper is novel, addresses a challenging question, and has detailed results. Answering a few questions will improve the paper's quality.
title: title: Official review
paper_summary: paper_summary: The paper proposes a framework to use a single diffusion-based policy to learn multiple locomotion tasks with pre-collected datasets of individual tasks, ranging from trotting, hopping, walking, pacing and running, across bipedal and quadrupedal robots. The proposed framework is tested on a real-world quadrupedal robot and demonstrates its efficiency in skill transition and robustness against different ground conditions.
main_review: main_review: **Strength:**
- The paper is generally well-written.
- The ability to transition between skills is very important, and to my best knowledge, quite novel. 
- Training from both bipedal robot data and quadrupedal robot data is important. The result of making a quadrupedal robot perform bipedal walking is compelling.
- The results on the robustness against different ground conditions is interesting.
- Making real-time inference possible for a relatively large transformer-based diffusion policy.
- Real robot experiments are a plus.

**Weaknesses:**
- The organisation of the sections can be somewhat improved. Quantitative results are a bit few in the main text. 
- Some parts can be explained more clearly in the main text, such as RHC framework. Additionally, the data collection for training are not clearly explained in the main text, which I think is quite important. 
- There are no quantitative results for bipedal robot experiments
- The names of the rows of the hyperparameters table (Appendix C.3) should match the notation in the paper for readability. For example, I assume “Predict Length” means n-step in the future action prediction?

**Minor notes:**
- I cannot seem to find the final K iteration for diffusion used in the proposed method.
- The bipedal robot experiments are done in simulation as far as I can gather. 
- Some citations can be added for better readability; such as AMP in section 6.
- Examples of the goals (commands) given for the shown skill transition would be nice.
questions: questions: 1. During data collection, how are goals (commands) determined during the collection of offline training datasets? Do the authors determine the goal for the locomotion tasks?

2. To my understanding, the proposed method implicitly learns the “transition” based on the goal. During training, how does the model determine which “goal” belongs to which “skill”? Is it given implicitly by the difference in goals for different skills? That is, are the goal difference between skills large enough such that the proposed method can know which “goal” belongs to which “skill” implicitly?

3. During evaluation, given a same goal, the paper mentioned that two skills might be deployed. How does the policy determine which skill to be the "default" one? As far as I understand, there are no additional information about the external disturbances during training. 

4. Following up, during evaluation, I wonder if it is possible to force the robot to perform certain skill? Given a goal, for example, the robot would want to hop. Can the proposed method force the robot to perform bouncing, for example?

5. Robustness to different grounds and terrains is very interesting. As far as I can gather, these information are not provided during training. Are these included in the data collection phase where the policies are trained using different ground and terrains?   

6. Somehow hard to quantify, but how smooth are the transition between skills? 

7. What is the reason for generating n-step future actions if only the next action will be used? Is it for the stability of training? (An ablation study of n would be nice to have).

8. What is the reason to use $a_{\text{t-h-2:t-2}}$ in line 173 instead of, for example, $a_{\text{t-h-1:t-1}}$ ? Can the authors elaborate?

9. Although not totally related, I wonder about the performance of the proposed method if it is only trained on a single skill dataset? That is, how much that learning a multi-skill policy harm single-skill performance?
limitations: limitations: 3
review_summary: review_summary: This paper proposes an interesting approach to tackle the problem of training a multi-skill policy using diffusion models. Overall the writing is clear the empirical results on real-world robots are interesting and I think would be a meaningful contribution to the community. I am leaning towards accepting this paper and I am willing to raise the score higher as long as the concerns listed above are addressed appropriately.
title: title: Review
paper_summary: paper_summary: This paper proposes to use diffusion models for imitation learning of complex multi-skill locomotion policies.
main_review: main_review: The authors follow the lines of prior work using diffusion models for imitation learning of manipulation skills. However, instead of using real-world datasets (e.g., from teleoperation), they use different sources from simulation, like training single-skill AMP policies. The authors show in various tasks and robots that their approach works well in simulation and on real hardware. 

### Strengths:
- First to implement diffusion imitation policies for locomotion
- The paper is clearly written.
- strong empirical results on real hardware.

### Weaknesses:
- Some parts of the experiments remain unclear (see questions?)
- There does not seem to be any novel insight compared to diffusion for manipulation (Only tricks like delays typically used for locomotion policies)
- hence, no algorithmic contribution.

Since this paper does not provide an interesting new insight into the diffusion model for locomotion but rather a straightforward extension of manipulation work, I believe that the most significant contribution of this paper to the research community is the code base. Since the authors, are promising to provide the latter upon acceptance, I vote for acceptance.
questions: questions: ### Questions:
- line 175: “Initiating inference before receiving the current state st allows parallel 175 processing and ensures up-to-date actions.” What exactly is processed in parallel?
- Line 184: “. Thus, we adopt DDPM for its en185 hanced robustness and performance in real-time control. “ What is adapted concretely? I didn’t find out even after the appendix? The appendix sounded more like a discussion?
- Line 189: “We answer these questions by presenting a state-of-the-art five190 skill policy that combines four quadrupedal skills, and more importantly, a bipedal locomotion skill 191 for quadrupedal robots, not yet demonstrated by prior RL frameworks.” In the related work are some works that did this already? → “[…] and walking on two feet with quadrupedal robots [20, 21, 22], requiring high agility and robustness. ”
- Section DiffuseLoco versus AMP (RL):
    - what exactly is a “skill”? Pacing and trotting ? Or each of them in a different direction and with a different velocity?
    - This is important to know, because it looks like your approach is surpassing AMP, which has been used to generate the dataset. So I guess that the AMP approach used to generate the dataset is a different one than used for comparison.
limitations: limitations: 3
review_summary: review_summary: See my summary above.