Adaptive Diffusion Terrain Generator for Autonomous Uneven Terrain Navigation

Authors: youwei  yu
junhong  xu
lantao  liu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-03 17:04:59

Abstract: Model-free reinforcement learning has emerged as a powerful method for developing robust robot control policies capable of navigating through complex and unstructured terrains.
The effectiveness of these methods hinges on two essential elements: 
(1) the use of massively parallel physics simulations to expedite policy training, 
and
(2) the deployment of an environment generator tasked with crafting terrains that are sufficiently challenging yet attainable, thereby facilitating continuous policy improvement. 
Existing methods of environment generation often rely on heuristics constrained by a set of parameters, limiting the diversity and realism.
In this work, we introduce the Adaptive Diffusion Terrain Generator (ADTG), a novel method that leverages Denoising Diffusion Probabilistic Models (DDPMs) to dynamically expand an existing training environment by adding more diverse and complex terrains tailored to the current policy.
Unlike conventional methods, ADTG adapts the terrain complexity and variety based on the evolving capabilities of the current policy.
This is achieved through two primary mechanisms:
First, by blending terrains from the initial dataset within their latent spaces using performance-informed weights, ADTG creates terrains that suitably challenge the policy. 
Secondly, by manipulating the initial noise in the diffusion process, ADTG seamlessly shifts between creating similar terrains for fine-tuning the current policy and entirely novel ones for expanding training diversity.
Our experiments show that the policy trained by ADTG outperforms both procedural generated and natural environments, along with popular navigation methods.

Main paper: 
 introduction: 
 Autonomous navigation across uneven terrains necessitates the development of control policies that exhibit both robustness and smooth interactions within challenging environments [1, 2, 3] . In this work, we specifically target the training of a control policy that allows the mobile-wheeled robots to adeptly navigate through diverse uneven terrains, such as off-road environments characterized by varying elevations, irregular surfaces, and obstacles.Recent advancements in reinforcement learning (RL) have shown promise in enhancing autonomous robot navigation on uneven terrains [4, 5, 6] . While an ideal scenario involves training an RL policy to operate seamlessly in all possible environments, the complexity of real-world scenarios makes it impractical to enumerate the entire spectrum of possibilities. Popular methods, including curriculum learning in simulation [7] and imitation learning using real-world collected data [8] , encounter limitations in terms of training data diversity and the human efforts required. Without sufficient data and training, the application of learned policies to dissimilar scenarios becomes challenging, thereby hindering efforts to bridge the train-to-real gap. Additionally, existing solutions, such as scalar traversability classification for motion sampling [9, 10] and optimization methods [11, 12] , may exhibit fragility due to sensor noise and complex characteristics of vehicle-terrain interactions. † Equal contribution. Demonstrations at https://adtg-sim-to-real.github.io.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. To tackle this challenge, we propose the Adaptive Diffusion Terrain Generator (ADTG), an environment generator designed to co-evolve with the policy, producing new environments that effectively push the boundaries of the policy's capabilities. Starting with an initial environment dataset, which may be from existing elevation data or terrains generated by generative models, ADTG is capable of expanding it into new and diverse environments. The significant contributions include:1. Adjustable Generation Difficulty: ADTG dynamically modulates the complexity of generated terrains by optimizing the initial noise (latent variable) of the diffusion model. It blends noisecorrupted terrains from the training environments, guided by weights derived from the current policy's performance. As a result, the reverse diffusion process, starting at the optimized initial noise, can synthesize terrains that offer the right level of challenge tailored to the policy's current capabilities. 2. Adjustable Generation Diversity: By adjusting the initial noise level before executing the DDPM reverse process, ADTG effectively varies between generating challenging terrains and introducing new terrain geometries. This capability is tailored according to the diversity present in the existing training dataset, enriching training environments as needed throughout the training process. Such diversity is crucial to ensure the trained policy to adapt and perform well in a range of previously unseen scenarios.We systematically validate the proposed ADTG framework by comparing it with established environment generation methods [13, 5] for training navigation policies on uneven terrains. Our experimental results indicate that ADTG offers enhanced generalization capabilities and faster convergence. Building on this core algorithm, we integrate ADTG with teacher-student distillation [14] and domain randomization [15] in physics and perception. We evaluate the deployment policy with zero-shot transfer to simulation and real-world experiments. The results reveal our framework's superiority over competing methods [16, 10, 4, 17] in key performance metrics. 
related work: 
 Offroad Navigation. Off-road navigation requires planners to handle more than simple planar motions. Simulating full terra-dynamics for complex, deformable surfaces like sand, mud, and snow is computationally intensive. Consequently, most model-based planners use simplified kinematics models for planning over uneven terrains [3, 18, 19, 20, 21] and incorporate semantic cost maps to evaluate traversability not accounted in the simplified model [2, 22] . Imitation learning (IL) methods [8, 23, 24] bypass terrain modeling by learning from expert demonstrations but require labor-intensive data collection. On the other hand, model-free RL does not require expert data and has shown impressive results enabling wheeled [7, 25, 4, 26] and legged robots [13, 5, 27, 28] traversing uneven terrains by training policies over diverse terrain geometries. However, the challenge is to generate realistic environments to bridge the sim-to-real gap. The commonly-used procedural generation methods [5, 13] are limited by parameterization and may not accurately reflect real-world terrain geometries. Our work addresses this by guiding a diffusion model trained on natural terrains to generate suitable terrain surfaces for training RL policies.Automatic Curriculum Learning and Environment Generation. Our method is a form of automatic curriculum learning [29, 30] , where it constructs increasingly challenging environments to train RL policies. While one primary goal of curriculum learning in RL is to expedite training efficiency [31, 32, 33] , recent work shows that such automatic curriculum can be a by-product of unsupervised environment design (UED) [34, 35, 36, 37, 38] . It aims to co-evolve the policy and an environment generator during training to achieve zero-shot transfer during deployment. Unlike prior works in UED, the environments generated by our method are grounded in realistic environment distribution learned by a diffusion model and guided by policy performance. Recently, a concurrent work proposes Grounded Curriculum Learning [39] . It uses a variational auto-encoder (VAE) to learn realistic tasks and co-evolve a parameterized teacher policy to control VAE-generated tasks using UED-style training. In contrast, our work uses a sampling-based optimization method to control the diffusion model's initial noise for guided generation.Controllable Generation with Diffusion Models. Controllable generation aims to guide a pretrained diffusion model to generate samples that are not only realistic but also satisfy specific criteria.A commonly used strategy is adding guided perturbations to modify the generation process of a pre-trained diffusion model using scores from the conditional diffusion [40, 41] or gradients of cost functions [42] . Another approach is to directly optimize the weights of a pre-trained diffusion model so that the generated samples optimize some objective function. By treating the diffusion generation process as a Markov Decision Process, model-free reinforcement learning has been used to finetune the weights of a pre-trained diffusion model [43, 44] . This approach can also be viewed as sampling from an un-normalized distribution, given a pre-trained diffusion model as a prior [45] .Our work is closely related to initial noise optimization techniques for guiding diffusion models [46, 47, 48] . Instead of refining the diffusion model directly, these methods focus on optimizing the initial noise input. By freezing the pre-trained diffusion model, we ensure that the generated samples remain consistent with the original data distribution. In contrast to existing approaches focusing on content generation, our work integrates reinforcement learning (RL) with guided diffusion to train generalizable robotic policies.Step Selection: The forward step k ∝ Λ -1 var is inversely proportional to the variance. We use a linear scheduler: k = K(1 -Λ var ), with K the maximum forward step and Λ var normalized to 0 ∼ 1. This inverse relationship ensures greater diversity in generated terrains. 3. Terrain Generation: Using the selected forward step k, apply our proposed Synthesize to generate new terrains, thus expanding variability for training environments. 
problem formulation: 
 We represent the terrain using a grid-based elevation map, denoted as e ∈ R W ×H , where W and H represent the width and height, respectively. This terrain representation is widely adopted in motion planning across uneven surfaces. Similar to most works in training RL policies for rough terrain navigation [13, 5] , we use existing high-performance physics simulators [49] to model the state transitions of the robot moving on uneven terrains s t+1 ∼ p(s t+1 |s t , a t , e). Here, s ∈ S and a ∈ A represent the robot's state and action, and each realization of the elevation (i.e., terrain) e specifies a unique environment. An optimal policy π(a|s, e; θ) can be found by maximizing the expected cumulative discounted reward. Formally,EQUATIONe∼p(e),st+1∼p(st+1|st,at,e) T t=0EQUATIONwhere p(s 0 ) is the initial state distribution and p(e) denotes the distribution over the environments. Due to the elevation e imposing constraints on the robot's movement, the policy optimized through Eq. ( 1) is inherently capable of avoiding hazards on elevated terrains. We aim to dynamically evolve the environment distribution p(e) based on the policy's performance, ensuring training efficiency and generating realistic terrain elevations. While constructing a realistic state transition p(s t+1 |s t , a t , e) is also important for reducing the sim-to-real gap, we leave it to our future work.A theoretically correct but impractical solution to Eq.( 1) is to train on all possible terrains Λ = (e 1 , ..., e N ), with p(e) as a uniform distribution over Λ. However, the vast variability of terrain geometries makes this unfeasible. Even if possible, it might produce excessively challenging or overly simple terrains, risking the learned policy to have poor performance [50] . Adaptive Curriculum Reinforcement Learning (ACRL) addresses these issues by dynamically updating the training dataset [51] . ACRL generates and selects environments that yield the most policy improvement. In our work, designing an effective environment generator is crucial. It should (1) generate realistic environments matching real-world distributions and (2) adequately challenge the current policy. Common approaches include using adjustable pre-defined terrain types [13] , which offers control but may lack diversity, and generative models [52] , which excel in realism but may struggle with precise policy-tailored challenges. In the following, we detail the proposed ADTG, which balances realism and policy-tailored terrain generation. 
adaptive diffusion terrain generator: 
 This section introduces the Adaptive Diffusion Terrain Generator (ADTG), a novel ACRL generator that manipulates the DDPM process based on current policy performance and dataset diversity. We begin by interpolating between "easy" and "difficult" terrains in the DDPM latent space to generate terrains that optimize policy training. Next, we modulate the initial noise input based on the training dataset's variance to enrich terrain diversity, fostering broader experiences and improving the policy's generalization across unseen terrains. We use e, e 0 , and e k to denote the environment in the training dataset, the generated terrain through DDPM, and the DDPM's latent variable at timestep k, respectively. All three variables are the same size R W ×H . Since in DDPM, noises and latent variables are the same [53] , we use them interchangeably.Latent Variable Synthesis for Controllable Generation. Once trained, DDPM can control sample generation by manipulating intermediate latent variables. In our context, the goal is to steer the generated terrain surface to maximize policy improvement after being trained on it. While there are numerous methods to guide the diffusion model [40, 44] , we choose to optimize the starting noise to control the final target [46] . This approach is both simple and effective, as it eliminates the need for perturbations across all reverse diffusion steps, as required in classifier-free guidance [40] , or fine-tuning of diffusion models [44] . Nevertheless, it still enhances the probability of sampling informative terrains tailored to the current policy.Consider a subset of terrain elevations Λ = (e 1 , e 2 , . . . , e n ) from the dataset Λ. To find an initial noise that generates a terrain maximizing the policy improvement, we first generate intermediate latent variables (noises) for each training environment in Λ at a forward diffusion time step k, e i k ∼ q(e i k |e i , k) for i = 1, ..., n. Assume that we have a weighting function w(e, π) that evaluates the performance improvement after training on each terrain map e i . We propose to find the optimized initial noise as a weighted interpolation of these latents, where the contribution of each latent e i k is given by the policy improvement in the original terrain environment w(e i , π).EQUATIONThe fused latent variable e ′ k is then processed through reverse diffusion, starting at time k to synthesize a new terrain e ′ 0 . The resulting terrain blends the high-level characteristics captured by the latent features of original terrains, proportionally influenced by their weights. We illustrate this blending effect in Fig. 1 . By controlling weights w i , we can steer the difficulty of the synthesized terrain.Weighting Function. Policy training requires dynamic weight assignment based on current policy performance. We define the following weighting function that penalizes terrains that are too easy or too difficult for the policy:EQUATION)It penalizes the deviation of terrain difficulty, s(e, π), experienced by the policy π from a desired difficulty level s. This desired level indicates a terrain difficulty that promotes the most significant improvement in the policy. The temperature parameter σ controls the sensitivity of the weighting function to deviations from this desired difficulty level. We use the navigation success rate [54] to represent s(•, •). While alternatives like TD-error [55] or regret [56] exist, this metric has proven to be an effective and computationally efficient indicator for quantifying an environment's potential to enhance policy performance in navigation and locomotion tasks [5, 13] . In Appendix A, we show that the optimized noise in Eq. ( 2) and the corresponding weighting function in Eq. ( 3) can be derived from formulating the noise optimization problem using Control as Inference [57, 58] and solving it through Importance Sampling. We denote the procedure of optimizing the noise e ′ k using Eq. 2 and generating the final optimized environment by reverse diffusion starting at e ′ k as e ′ = Synthesize( Λ, π, k), where k is the starting time step of the reverse process. As discussed in the next section, a large k is crucial to maintaining diversity.The preceding section describes how policy performance guides DDPM in generating terrains that challenge the current policy's capabilities. As training progresses, the pool of challenging terrains diminishes, leading to a point where each terrain no longer provides significant improvement for the policy. Simply fusing these less challenging terrains does not create more complex scenarios. Without enhancing terrain diversity, the potential for policy improvement plateaus. To overcome this, it is essential to shift the focus of terrain generation towards increasing diversity. DDPM's reverse process generally starts from a pre-defined forward step, where the latent variable is usually pure Gaussian noise. However, it can also start from any forward step K with sampled noise as e K ∼ q(e K |e 0 ) [59] . Fig. 1 shows the variance of generated terrains decreases with fewer forward steps and vice versa. To enrich our training dataset's diversity, we propose the following: 1. Variability Assessment: Compute the dataset's variability Λ var by analyzing the variance of the first few principal components from a Principal Component Analysis (PCA) on each elevation map. This serves as an efficient proxy for variability.Input: Pretrained DDPM ϵ(•, •; ϕ), an initial terrain dataset Λ Output: The optimized privileged policy π * Initialize: The privileged policy π 1: while π not converge do k = K(1 -Λ var ) ▷ Sec. 4.2 5: e ′ 0 = Synthesize(Λ, π, k) ▷ Sec. 4.1 6: Λ ← Λ ∪ e ′ 0 ▷ Update Dataset 7: end whileWe present the final method pseudo-coded in Alg. 1 using the proposed ADTG for training a privileged policy. The algorithm iterates over policy optimization and guided terrain generation, co-evolving the policy and terrain dataset until convergence. The algorithm starts by selecting a training environment that provides the best training signal for the current policy, which can be done in various ways [50] . For example, one can compute scores for terrains based on the weighting function in Eq. ( 3) and choose the one with the maximum weight. Instead of choosing deterministically, we sample the terrain based on their corresponding weights. The Optim step collects trajectories and performs one policy update in the selected terrain. After the update, we evolve the current dataset by generating a new one, as shown in lines 4 -6 of Alg. 1. In practice, we run Alg. 1 in parallel across N terrains, each with multiple robots. In parallel training, Synthesize begins by sampling N × n initial noises, where N is the number of new terrains (equal to the number of parallel environments) and n is the sample size in Eq. ( 2). It then optimizes over these noises to generate N optimized noises. Finally, these optimized noises are passed to the DDPM to generate N terrains. When the dataset grows large, it sub-samples terrains from Selector's complement, with success rates updated by the current policy. We validate effectiveness of Selector in Sec. 5.1 and Synthesize in Appendix A.3, which also explains its sub-sampling logic.ADTG with Policy Distillation. While ADTG can train policies to generalize over terrain geometries, real-world deployments face challenges beyond geometry. This includes noisy, partial observations and varying physical properties. To address these, we distill a privileged policy, which observes the complete elevation map and noiseless states into a depth vision-based policy with noisy measurements. The privileged policy is trained using PPO [60] , and the deployment policy is trained using DAgger [61] , utilizing methods similar to [6] . To enhance generalization, we integrate physical and perception domain randomization [15] (more details in Appendix. C). The training loop integrating ADTG with teacher-student distillation is illustrated in Fig. 2 and the Appendix. With ADTG validated in Sec. 5.1, our whole system is demonstrated in the sim-to-deploy experiments. 
experiments: 
 In this section, we start with the algorithmic evaluation to highlight the effectiveness of our ADTG. Then through sim-to-sim on two kinds of wheeled robot platforms and sim-to-real on one wheeled and one quadruped robot, we study the performance of our ADTG policy against ablations and competing methods.We train in IsaacGym [49] and parallel 100 uneven terrains, each with 100 robots. Among the elevation dataset [62] as detailed in Appendix C.1, we select 3000 for DDPM training (E-3K), 100 for algorithmic evaluation (E-1H), and 30 for sim-to-sim experiments (E-30). Simulations run on an Intel i9-14900KF CPU and NVIDIA RTX 4090 GPU. Real-world tests use an NVIDIA Jetson Orin. This section evaluates whether the environment curriculum generated by ADTG enhances the generalization capability of the trained privileged policy across unfamiliar terrain geometries, on the wheeled ClearPath Jackal robot. We compare with the following baselines. Procedural Generation Curriculum (PGC), a commonly used method, uses heuristically designed terrain parameters [13] . Our implemented PGC follows ADTG, adapting the terrain via the score function Eq. ( 3) and dynamically updating the dataset. First, to ablate our Adaptive curriculum, Diffusion Terrain Generator (ADTG) generates terrain without curriculum. Procedural Generation (PGC) randomly samples parameters. To ablate our Diffusion Generator, Natural Adaptive Terrain (N-ADTG) selects terrains directly from E-3K.To ablate both, Natural Terrain (N-ADTG) randomly samples from E-3K without curriculum. Mono font means the ablated parts. Second, to test ADTG's robustness to initialization other than diffusion, N-ADTG and P-ADTG start with datasets sub-sampled from E-3K and procedural generations.All methods use the same training and evaluation setup. After each training epoch, policies are tested in held-out evaluation environments with 200 000 start-goal pairs. Fig. 3 shows the normalized RL return 1 . ADTG outperforms PGC and N-AT within 40 epochs, exceeding the return by over 0. | , where ω and a denote angular velocity and linear acceleration. These motion stability indicators are crucial in mitigating sudden pose changes and enhancing overall safety. The trajectory ratio is the successful path length relative to straight-line distance and indicates navigator efficiency. All baselines use the elevation map [63] with depth camera and identify terrains as obstacles if the slope estimated from the elevation map exceeds 20 • . For orientation costs, we obtain the robot's roll and pitch by projecting its base to the elevation map.Baselines. Falco [16] , a classic motion primitives planner, and MPPI [10, 64] , a sampling-based model predictive controller, are recognized for the success rate and efficiency. They use the pointcloud and elevation map to weigh collision risk and orientation penalty. TERP [4] , an RL policy trained in simulation, conditions on the elevation map, rewarding motion stability and penalizing steep slopes. POVNav [17] performs Pareto-optimal navigation by identifying sub-goals in segmented images [65] , excelling in unstructured outdoor environments.Simulation Experiment. We simulate wheeled robots, ClearPath Jackal and Husky, in ROS Gazebo on 30 diverse environments (E-30), equipped with a RealSense D435i camera. We add Gaussian noises to the robot state, depth measurement, and vehicle control to introduce uncertainty. 1000 start and goal pairs are sampled for each environment. We do not include ablations other than N-AT because of poor algorithmic performance. As Jackal's results shown in Table 1 , our method outperforms the baselines. Appendix C.2 provides statistical results for Husky. While all methods show improved performance due to the Husky's better navigability on uneven terrains, our method consistently outperformed baseline methods. The depth measurement noise poses a substantial challenge in accurately modeling obstacles and complex terrains. Falco and MPPI often cause the robot to get stuck or topple over, and TERP often predicts erratic waypoints that either violate safety on elevation map or are overly conservative. Learning-based TERP and POVN lack generalizability, with their performance varying across different environments. This issue is mirrored in N-AT and PGC, highlighting the success of adaptive curriculum and realistic terrain generation properties of ADTG.Real-world Experiment. The Jackal robot is equipped with a Velodyne-16 LiDAR, a RealSense D435i camera, and a 3DM-GX5-25 IMU. We test in 9 diverse representative environments, grass, forest, arid, rust, mud, gravel, dune-hard, dune-tough, and dune-expert, displayed in Fig. 4 . For each environment, we sample 36 start-goal pairs. Note that the results of N-AT and PGC policies are not presented as they achieved limited success in real-world experiments. Table 1 provides an overview of the performance metrics, with success rates across all methods de- tailed in the Appendix C.3. In forest, POVNav struggled with image segmentation of complex ground elements like foliage and underbrush. Other baselines encountered difficulties due to imprecise depth data affecting terrain modeling. All baselines risked getting stuck in muddy, sandy, and rocky areas. In contrast, our method autonomously engaged in arc movements, avoiding immobilization in slippery environments. In dunes, characterized by steep elevation changes, failures of other methods were due to incorrect obstacle detection or severe slope changes causing overturn.In addition, we ablated Physics and Perception domain randomization to study their contributions.Removing either reduced success rates, though both ablations outperformed baselines due to ADTG. The perception domain suffered from camera depth issues tied to exposure. Without the physics domain, the robot showed improved orientation smoothness but struggled to make reactive behaviors, since the learned arc movements in slippery areas benefited from this domain.Real-World Quadruped Locomotion. To validate ADTG's generalization across different embodiments, we benchmark against PGC and the built-in MPC controller on the Unitree Go1 quadruped in the same environments as Jackal. Both ADTG and PGC were trained using the Parkour [66] "walk" policy. Similar to the hiking experiment [5, 13] , we conducted a continuous 1.2 km loop, following MPC's footprints for fairness. Failures, defined as rollovers, occurred when detours around challenging areas were not allowed. MPC had 6 rollovers 3 from high-speed commands, 3 on challenging gravel and dune terrains). ADTG policy exhibited natural postures with 2 failures due to high-speed commands. PGC policy showed robust prostrate postures but 11 failures due to sudden movements (jumps) on all terrains except grass. The results highlight ADTG's advantages in learning robust behavior. See the appendix for the full Jackal ablation analysis and quadruped experiment details. 
conclusion, limitations and future directions: 
 We propose an Adaptive Diffusion Terrain Generator (ADTG) to create realistic and diverse terrains based on evolving policy performance, enhancing RL policy's generalization and learning efficiency.To guide the diffusion model generation process, we propose optimizing the initial noises based on the potential improvements of the policy after being trained on the environment denoised from this initial noise. Algorithmic performance shows ADTG's superiority in generating challenging but suitable environments over established methods such as commonly used procedural generation curriculum. Combined with domain randomization in a teacher-student framework, it trains a robust deployment policy for zero-shot transfer to new, unseen terrains. Extensive sim-to-deploy tests with wheeled and quadruped robots validate our approach against SOTA planning methods.Limitations and Future Directions: A key limitation of ADTG is that it evolves only the environment distribution, relying on physics simulators for state transitions, limiting deployment in the complex real world. While domain randomization helps, it's not a full solution. Future work will integrate environment distribution and physics to bridge the sim-to-real gap. Additionally, ADTG's environment scale is suited for local planning, but larger environments are needed for long-horizon tasks. We plan to explore hierarchical diffusion models for generating multi-layered environments.where t ∈ [1, k]. Our goal is to find the noise that maximizes the Eq. ( 5), e * k = arg max e k J (e k ). Gradient-based methods using differentiable rewards [4] are challenging to apply in our setup, as computing the gradient of policy improvement with respect to the initial noise is impractical. This is because evaluating policy improvement requires simulating the policy on each generated terrain to compute success rates. Instead, we frame this as a control-as-inference problem, using approximate sampling to optimize the initial noise for Eq. ( 5). Following the KL control theory derivation [1] , we aim to find a distribution of the starting noise q(e k ) that optimizes Eq. ( 5) while remaining close to a reference noise distribution q(e k ). This is achieved by adding an extra KL cost to the return in Eq. ( 5) and taking expectation with respect to the random initial noiseEQUATION= q(e k ) J (e k ) -log q(e k ) q(e k ) de k = q(e k ) log (exp {J (e k )}) -log q(e k ) q(e k ) de k = -q(e k ) log q(e k ) q(e k ) exp {J (e k )} de k = -KL (q(e k )∥ψ(e k )) ,where ψ(e k ) ∝ q(e k ) exp J (e k ) is an unnormalized distribution with high density at the high return region. The Kullback-Leibler (KL) divergence is non-negative, implying that the optimal return is achieved when q = ψ. Consequently, sampling from ψ is equivalent to sampling from the optimal noise distribution. While there are multiple methods to sample from ψ, we employ importance sampling to compute the expected value of the noise. This approach is chosen due to its simple computational structure within this formulation. The importance sampling can then be expressed asE [e k |e k ∼ ψ(e k )] = 1 Z e k q(e k ) exp {J (e k )} de k (7) = 1 Z E [exp {J (e k )} | e k ∼ q(e k )] ≈ 1 Z 1 N N i=1 exp {J (e k )} e k ,where the samples e k are drawn from the reference distribution. The normalization constant can be estimated similarlyZ = q(e k ) exp {J (e k )} de k (8) = E [exp {J (e k )} |e k ∼ q(e k )] ≈ 1 N N i=1exp {J (e k )} .Combining Eq.( 7) and Eq.( 8), we obtain the expected optimal noiseEQUATION)We define the reference distribution as the marginal distribution of the forward diffusion process q(e k ) = q(e k |e)p(e)de, where p(e) is the distribution over the current training environment dataset. Sampling from q(e k ) involves drawing from p(e) and then from q(e k |e), which is exactly the process described in Section 4.1. This process is efficient because the forward diffusion q(e k |e) is easy to sample, and we can readily draw samples from p(e) using the available environment dataset. The steps in our algorithm directly correspond to the computation of the final expected noise as expressed in Eq. ( 7). Table 2 illustrates the mapping between these algorithmic steps and the computational steps of computing Eq. ( 7). Importance sampling solver (Eq. ( 7)) Weighted interpolation (Eq. ( 2))Table 2 : Correspondence of algorithmic steps in ADTG with control as inference using importance sampling. 
 
 Appendix: The core assumption underlying our method is that the success rates of the generated terrains by ADTG are consistent with the weighted combination of the success rates of selected terrains. This appendix provides empirical evidence to support this assumption. During training, we sample terrains with the success rate outside the 0.6 to 0.85 range for Synthesize. As our dataset expands rapidly, we uniformly sub-sample up to 1000 terrains as the current (sub-)dataset. If the synthesized terrain's predicted success rate falls outside the 0.6 to 0.85 range, we adjust by adding suitable samples while ensuring the total number under 16. We choose 16 based on the GPU memory limitation. We first set aside 100 terrains as dataset D and assess their success rates using the privileged policy. For each synthesized terrain, Selector sub-samples N terrains from D, where N follows the curriculum. For each synthesized terrain, we define the predicted success rate ass ′ = ( n i=1 w i s i )/( n i=1 w i ), where w and s denote Synthesize weight and success rate, respectively. The validation involves comparing the actual success rates evaluated by the privileged policy against the predicted success rates. Fig. 5 illustrates this comparison across 8 diffusion steps, with each step containing 60 synthesized samples. For clarity, each step's display is organized into two rows. The first row presents the predicted success rates, and the second row shows the actual success rates. The similarity in color between these two rows indicates the consistency of the success rates for the generated terrains. Our result shows that most of the samples closely align with the predicted success rates. The privileged (teacher) policy is trained using Proximal Policy Optimization (PPO) [7] for goaloriented navigation on uneven terrains, minimizing motion vibration and jerk. The design is as following:State, Observation, and Action. The state s tr t = [q t , v t , ω t , d t , a θ t-1 ] at timestamp t is ground-truth motion information, including the orientation in quaternion q t ∈ R 4 , linear velocity v t ∈ R 3 , angular velocity ω t ∈ R 3 , relative goal distance d t ∈ R 2 , and previous action a θ t-1 ∈ R 2 . The reference frame is anchored to the robot base. Normalization is applied to linear and angular velocities. We refrain from normalizing the relative goal distance, as this allows the policy to adapt to varying goal range scales. The observation has full access to the elevation map o tr ∈ R 128×128 . Each observation remains constant during an ACRL episode when ADTG does not evolve terrains. The action a tr t = [α v v x t , α ω ω z t ] applied to the robot represents proportional-derivative (PD) targets for forward linear and yaw angular velocities, where a θ t = [v x t , ω z t ] is the network output, and [α v , α ω ] are coefficients for the vehicle drive system.Rewards. The total reward is structured as a sum of the weighted components: Goal Proximity c 1 1 δg (s t ) assesses the robot's closeness to the goal against thresholds δ pos and δ rot , Orientation Regulation c 2 1 δe (E(q t )) imposes penalties for exceeding safe roll and pitch angles, Movement Consistency c 3 ||a t -a t-1 || 2 + c 4 || ȧtȧt-1 || 2 incentives consistent and gradual actions, Groundcontact and Safety c 5 dim(f = 0) + c 6 dim(f > δ F ) discourages situations where the contact force mean ground contact loss or a collision risk. The total reward is parameterized as follows:EQUATIONStart and Termination. Episodes start with robots in random poses and assigned navigation targets. The initial state s 0 is sampled by position, velocity, and yaw, with other parameters managed by the physics engine. The state z considers terrain geometry to avoid immediate failure. Initially, goals are within a circular sector, with a radius of d max = [0.5, 3.0]m and a central angle of [-π/3, π/3]rad.As the policy progresses to success rate of 60%, the radius and angle incrementally increase by 0.5m and π/18 until a semicircle with a radius of 6m. An episode ends upon reaching the goal or triggering safety or running length constraints. The maximum episode length is d max /∆t/v avg , with v avg = 0.3 m/s the minimum average velocity and ∆t = 0.005 s the simulator timestep. Early termination occurs if the roll or pitch angle exceeds δ quat = π/9, or if the robot collides with the environment, or if two or more wheels are off the ground, risking toppling or entrapment.The deployment (student) policy θ is trained via Dataset Aggregation (DAgger) [8] Perception Domain Randomization. Since velocity relies on the inertial measurement unit (IMU), which is typically noisy [14] , we add Gaussian noise to simulate this: ṽ ∼ N (v, σ2 v ) for linear velocity and ω ∼ N (ω, σ 2 ω ) for angular velocity, where v and ω are ground-truth in the simulator. Based on the empirical study [15] , depth cameras suffer from precision and lateral noise, as well as invalid values at full sensor resolution (nan ratio). We model precision noise as N (0, p 0 +p 1 d+p 2 d 2 ) and lateral noise as N (0, l 0 + l 1 • θ/(π/2 -θ)), where d is the measured depth of pixel (u, v) and θ is its azimuth. The nan ratio is modeled by a uniform distribution U(0, δ nan ).During training, the IsaacGym simulator runs at 200 Hz, and the policy runs at 50 Hz. In simulation experiments, ROS Gazebo provides odometry at 1000 Hz. In real-world experiments with Jackal, Faster-LIO [13] with IMU preintegration [14] fuses Lidar and IMU for 200 Hz odometry. Both settings use a depth camera RealSense D435i with [640 × 480] resolution at 30 fps, synchronized to odometry using ROS approximate time.Uneven Terrain Datasets. To generate datasets, we utilize a digital elevation model (DEM) represented as a raster obtained from a real-world high-resolution topography dataset 2 . The entire map is seamlessly divided into tiles of size [128 × 128] with resolution 0.1 m. 3000 training data for DDPM and 100 evaluation data for algorithmic performance are randomly selected within a specific geographical range, defined by latitude and longitude intervals: [33.5874, -116 Policy Parameterization.Our privileged (teacher) and deployment (student) policies use an encoder-decoder architecture. The encoder utilizes the first 16 layers of ResNet-18 [9] to extract feature representations from the terrain elevation or depth image. The deployment policy employs a neural circuit policy (NCP) [10] , specifically the closed-form continuous-time (CfC) network, to generate linear and angular commands. While Vanilla RNN, LSTM, GRU, and NCP achieve similar accuracy with sufficient learning steps, we choose NCP for its superior performance with significantly fewer parameters [10] . We show details of PPO in Table 3, networks in Fig. 6 , and domain randomization in Table 4 . This setting keeps consistent for sim-to-sim and sim-to-real experiments. environment generates terrains based on the trigonometric functions. Similarly, other procedurally generated environments are also navigable.In addition to benchmarking on the ClearPath Jackal robot, we train the ClearPath Husky robot using the same settings as the Jackal and tested in the same ROS Gazebo environments E-30 detailed in Sec. C.1. We evaluated across 30 environments, each containing 1000 pre-sampled start-goal pairs, resulting in total 30 000 trials. The results in Table 5 demonstrate that our method generalizes effectively across different wheeled platforms, maintaining an advantage over other methods in terms of success rate, orientation vibration, and position jerk. All methods performed better overall with the Husky, due to its better navigability on uneven terrains than Jackal. However, the performance differences between our method, Natural Adaptive Terrain (N-AT), and Procedural Generation Curriculum (PGC) relative to other methods mirror those observed in the Jackal experiment, with challenges such as perception noise and erratic prediction persisting for competing methods. In summary, our method generalizes well across different wheeled platforms and maintains a performance advantage over baseline methods. Wheeled Robot Navigation. In the Jackal real-world experiment, we ablate the physics (w/o Physics) and perception (w/o Percept) domain randomization (DR) to study their contributions, shown in Table 6 and Table 7 . We observed varying contributions of physics and perception domain randomization. First, in forest, mud, and gravel, removing either domain decreased the success rate, but these ablations still performed better than baselines due to ADTG. In arid and rust, the perception domain was crucial because of increased perception noise. Second, we identified drawbacks in the perception domain, particularly in dunes where the camera depth quality heavily depended on exposure, which is difficult to model. Last, without the physics domain, the robot showed improved orientation smoothness but struggled to make reactive behaviors, since the learned arc movements in slippery areas benefited from this domain. Additionally, for computational efficiency per frame, our method averaged 28.68 ms over 10 The issue of imprecise depth measurements is highlighted in Fig. 7 , where portions of the elevation map appear above the trajectories due to depth measurements inaccurately portraying unstructured objects. Compared to baselines that rely on elevation maps, our method showed more success. This is because the perception randomization during training introduces depth noises, allowing the Table 7 : Detailed results of success rate among Falco, MPPI, TERP, POVNav, and our method with ablations of physics and perception domain randomization. Each environment for each method has 36 start-goal pairs. policy to be more robust against misleading depth measurements. Similar results are also observed in a recent work for legged locomotion [5] . The complex vehicle-terrain interactions posed another challenge. As shown in Fig. 7 , our method's trajectories had more curves, representing reactive behavior in slippery areas and leading to the relatively large trajectory ratio in Table 1 . The learned arc movements are simple but effective, which will be explored further in our future research.Quadruped Robot Locomotion. Refer to Parkour [12] Appendix B for detailed training in simulation. The parameters that differ from Parkour are as follows, with all other settings remaining the same. We set the maximum iteration of PPO to 1 × 10 5 and the number of parallel environments to 200. For the environment, the terrain size is [128 × 128] with a horizontal scale of 0.1. The nine experimental environments with various challenges, including slippery surfaces, sloped terrains, and unpredictable ravines. These environments are the same as those used in the ground vehicle experiment but were traversed continuously in a loop. The gaits of our ADTG and the built-in Go1 MPC were natural, unlike the Procedural Generation Curriculum (PGC), which showed robust but prostrate postures. The PGC policy performed well but faced limitations due to sudden movements (jumps) that caused the robot to roll over, as shown in Fig. 8 . PGC's issue may originate from its over-challenging unrealistic environments and we propose to investigate in the future. It is calculated by the actual return divided by value function's upper bound based on our reward function.https://opentopography.org 


Decision: This paper has been reasonably well-received by the reviewers.  The paper is relatively clear and well-motivated.  

The reviewers appreciated the diversity of different platforms this was tested on (with the exception of reviewer 2 pointing out a single wheeled platform - more wheeled platforms would be appreciated).

The comparison to other navigation algorithms, and the use of real robots for testing are other strong points.

Experimentation is generally good, with a number of different studies undertaken.

I have some reservations about how scalable this is - in that the terrain maps used are presumably of a given feature size, and thus would only produce maps of a similar feature size.  Can the authors please comment?

The testing section is weak.
-There is a lack of specifics of the testing and evaluation - e.g., what is a failure? Some of the experimental and testing settings are missing (see reviewer 2)

-There is a lack of forthright discussion on limitations

I encourage the authors to fully address the reviewer comments to increase chances of acceptance.  Specifically, addressing limitations, adding more detail to the experimental session, and showing more relevant information will help to increase scientific rigor.


Update:

The authors have responded well and improved their submission in response to reviewer comments.

Reviews: review: title: title: Terrain Generation could be a potential good topic for robustness, the algorithm design lacks further justification.
paper_summary: paper_summary: The paper introduces the Adaptive Diffusion Terrain Generator (ADTG), a novel environment generation method for training autonomous robots to navigate uneven terrains. This method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to create diverse and challenging terrains tailored to the robot's current capabilities. Unlike conventional heuristic-based methods, ADTG dynamically adjusts terrain complexity and variety based on the robot's evolving performance.
main_review: main_review: The key contributions of this work include:
 ADTG synthesizes terrains of varying difficulties by blending terrains from an initial dataset within their latent spaces, guided by performance-informed weights.
 By manipulating the initial noise in the diffusion process, ADTG shifts between creating similar terrains for fine-tuning and entirely novel ones for expanding training diversity.
 Experiments demonstrate that policies trained with ADTG outperform those trained in procedural or natural environments, showing enhanced generalization capabilities and faster convergence.
 ADTG integrates with adaptive curriculum reinforcement learning (ACRL), co-evolving the terrain dataset and the robot's policy to continuously challenge and improve the robot's capabilities.
The proposed method addresses limitations in existing terrain generation techniques by providing a more diverse and realistic training environment, thereby enhancing the robustness and adaptability of autonomous navigation policies.
questions: questions: 1. Eq.1, the definition is not clear. The expectation is over what?(Policy or terrain parameters?)
2. A deeper explanation of the design choices in ADTG, such as the selection of DDPMs over other generative models, and the specific weighting function used for blending terrains, would provide a better understanding of the underlying principles and their impact on performance.
3. While the experiments demonstrate the effectiveness of ADTG, additional real-world validation across a broader range of scenarios and environments could further substantiate the claims.
limitations: limitations: 2
review_summary: review_summary: While the Adaptive Diffusion Terrain Generator (ADTG) presents an innovative approach to terrain generation, there are areas where the justification of the algorithm design could be strengthened.
title: title: Review
paper_summary: paper_summary: This paper introduces Adaptive Diffusion Terrain Generator (ADTG), a two-stage approach to generating terrain maps for training policies in simulation (IsaacGym) for effective learning and robust deployment to other simulation (Gazebo), and real world. The first stage is to train a Denoising Diffusion Probabilistic Model from 3000 representative terrain maps based on real terrain data, and using this model to blend terrains from the initial dataset with generated maps with a weighting determined from policy success. The second is to add more diverse and complex terrains by manipulating the noise in the diffusion process.
main_review: main_review: This paper is well written and ideas are presented clearly. The results are thorough, with significant ablations and comparisons. Extensive experiments across simulations, and diverse terrain types and robot embodiment's in challenging real world scenarios validate the robustness of this method compared to several baselines. The work from this paper offers an exciting terrain generation method and curriculum-based learning approach that will clearly benefit the robot learning community. 

I couldn't see an explicit report of limitations, I'm not sure if this is still a requirement for this conference.

Minor feedback:

In Section 4.1 and 4.2, ‘e’ is used for the DPPM latent variable and for individual terrain maps. Are they the same size, i.e a latent variable for each pixel of a terrain map and corresponding weighting term? Adding some clarity will help distinguish between these variables.

What is “trajectory ratio” in Table 1? 

In Algorithm 1, is the “Optim” step for a single rollout, or to convergence for a given terrain map? How many terrains are trained in parallel, and does each robot have a different terrain instance? 

For each Policy Update a new terrain is added, the number of terrains in Lambda would quickly grow, is the variance calculation of 4.2 performed over the entire set of terrains? How long does it take to run through each stage in Algorithm 1, what's the most compute/time intensive step, and the overall training time?

“we propose the follows:” -> “we propose the following:”
questions: questions: See above
limitations: limitations: 2
review_summary: review_summary: Clearly a standout work with novel contributions and significant experimental results on hardware.
title: title: Quality paper advancing wheeled robots' ability to traverse rough terrain.
paper_summary: paper_summary: This paper presents a diffusion-based terrain generation and assignment algorithm. The author's theorise that by providing a terrain generator that is both realistic and flexible will improve policy performance when trained with reinforcement learning.  They ablate their methods with existing work that uses parameterised terrains (e.g. steps, perlin noise), and show that non-parametric diffusion generators outperform the parameterised methods. They further ablate their methods with existing navigation methods, and do extensive real world training. They propose a blending strategy to blend the latent space of two (or more?) terrain embeddings to get new terrains with varied traverse difficulties. They also propose adjusting the diffusion denoising steps to gradually increase diversity over time.
main_review: main_review: The paper is presented well and covers most important points to justify that the proposed method is an improvement on SOTA. 

Strengths:

- combines diversity and difficulty adjustment intuitively into the diffusion framework

- the task of uneven terrain nav w/ wheeled robots is an important area for robotics

- Their results show a substantial improvement (10-20% success rate improvements)

- Quality of implementation is good and figures are very illustrative

- Tested on a real platform in a variety of environments (e.g. grass, forest, gravel, sand)

- Blending approach is verified as an approximation of terrain difficulty in appendix

Weaknesses:

- Only one wheeled platform is tested (Jackal). There may be some variability with other wheeled configurations

- A quadruped (Unitree Go1) is tested, while the author's method seems to perform better, the results are very limited (no tables presented, only mentions number of failures but some experimental details are missing (e.g. the time the robots were tested in each experiment)

- The main baseline of this paper is the Procedural Content Generation (PCG) (reference 13) which was intended for quadrupeds. The terrain generation parameters may not be appropriate for a wheeled robot, e.g. the author includes steps of heights 0.1-0.3m, which may be impossible for this robot and hinder the performance of the baseline due to the discrepancy in robot morphology

- No limitations are discussed explicitly in this paper
questions: questions: 1. Are the PCG parameters and terrain types appropriate for the Jackal? Are you able to verify this or ablate them? If the ADTG policy can't traverse a reasonable range of these PCG terrain parameters then these should be adjusted for fair comparison.  

2. Are there limitations of your proposed method?

3. Were the quadruped experiments ran in the same was as the Jackal experiments (sampling 36 start-goal pairs per experiment)?
limitations: limitations: 1
review_summary: review_summary: If the authors provide a justification for the baseline (PCG) parameters used or an ablation to show these are not unfairly disadvantaging the baseline, I would strongly recommend this paper for CoRL. Without this justification, I am uncertain but leaning towards 'accept' as many non-curriculum based baselines are compared and the contributions are valuable nonetheless.
title: title: Review of Submission166
paper_summary: paper_summary: This paper presents a model-free reinforcement learning approach proposing robust robot control policies for navigating complex terrains, relying on parallel physics simulations and environment generators. The Adaptive Diffusion Terrain Generator (ADTG) presented here addresses the limitations of traditional approaches of parameter-based heuristics by using Denoising Diffusion Probabilistic Models (DDPMs) to create diverse, complex terrains tailored to the policy's abilities. ADTG blends terrains with performance-informed weights and manipulates initial noise to generate suitable challenges and novel environments. Experimental results claim that ADTG-trained policies outperform those trained with traditional methods and natural environments.
main_review: main_review: The writing is clear and the paper provides insights into terrain generation in software environment via reinforcement learning. 
The authors claim that their terrain generation approach can capture the contact dynamics of very complex terrains and they try to show their method's applicability via simulations and robot experiments. Although the context is original and highly relevant to the robotics community, as it attempts to address the significant challenge of bridging the sim-to-real gap, I have concerns regarding the feasibility and scientific rigor of their methods. Specifically, capturing the terra-dynamics of complex, deformable, and flowable surfaces (such as sand, mud, and snow) is a very challenging task due to the highly nonlinear interactions of millions of particles during robotic limb contact. Also such interactions are bound to hysteresis effects, i.e., the terra-dynamics are different if the robot encounters a previously disturbed terrain. Researchers often attempt to capture these interactions using experimental methods, which only partially work and rely on various assumptions. See the details of such an attempt in the paper below:

Li et al, A terradynamics of legged locomotion on granular media, 2013

Although the proposed approach claims to outperform other methods, it is unclear whether this performance increase is due to the proposed control policy itself or other parameters related to robot-terrain interactions. And such a task would require extensive investigations of granular physics which I believe is out of the scope for CORL.

Strengths:
Model-free reinforcement learning is highly effective for developing robust robot control policies for navigating complex terrains.

The use of massively parallel physics simulations speeds up policy training significantly.

Weaknesses:
There are concerns regarding the feasibility and scientific rigor of the proposed methods.

Capturing the terra-dynamics of complex, deformable, and flowable surfaces (such as sand, mud, and snow) is highly challenging, and current methods only partially work under various assumptions.

The evaluation of the robots' performance is not clearly presented.
questions: questions: Fig 1, fig 4 and fig 6 are small hard to grasp.

It is unclear how section 4 generates flowable and deformable surfaces such as sand and mud.

Section 5.2:
The metrics are not clearly explained.
The details regarding the robot's parkours in both sim and exp subsections are not given. Also, I realized that the mud terrain does not seem to be wet in the video. 
Limitations are not addressed. 
The conclusion section does not seem to reflect the key messages of the paper.
limitations: limitations: 1
review_summary: review_summary: I have concerns regarding the feasibility and scientific rigor of their methods, particularly in capturing the terra-dynamics of complex, deformable, and flowable surfaces like sand and mud. These interactions are nonlinear and influenced by hysteresis effects, making accurate predictions challenging. While the authors claim their method outperforms existing ones, it is unclear if this is due to the proposed control policy or other factors related to robot-terrain interactions. This area requires extensive investigations of granular physics, which may be beyond the scope of this work.