Adaptive Diffusion Terrain Generator for Autonomous Uneven Terrain Navigation
Authors: Youwei Yu
Junhong Xu
Lantao Liu
Venue: CoRL 2024 (2024)
Keywords: Curriculum Reinforcement Learning
Diffusion Model
Field Robotics
TL;DR: Our work introduces Adaptive Diffusion Terrain Generator, leveraging DDPMs to dynamically generate complex, realistic terrains for improved policy training and generalization to real-world environments.

Abstract: Model-free reinforcement learning has emerged as a powerful method for developing robust robot control policies capable of navigating through complex and unstructured terrains.
The effectiveness of these methods hinges on two essential elements: 
(1) the use of massively parallel physics simulations to expedite policy training, 
and
(2) the deployment of an environment generator tasked with crafting terrains that are sufficiently challenging yet attainable, thereby facilitating continuous policy improvement. 
Existing methods of environment generation often rely on heuristics constrained by a set of parameters, limiting the diversity and realism.
In this work, we introduce the Adaptive Diffusion Terrain Generator (ADTG), a novel method that leverages Denoising Diffusion Probabilistic Models (DDPMs) to dynamically expand an existing training environment by adding more diverse and complex terrains tailored to the current policy.
Unlike conventional methods, ADTG adapts the terrain complexity and variety based on the evolving capabilities of the current policy.
This is achieved through two primary mechanisms:
First, by blending terrains from the initial dataset within their latent spaces using performance-informed weights, ADTG creates terrains that suitably challenge the policy. 
Secondly, by manipulating the initial noise in the diffusion process, ADTG seamlessly shifts between creating similar terrains for fine-tuning the current policy and entirely novel ones for expanding training diversity.
Our experiments show that the policy trained by ADTG outperforms both procedural generated and natural environments, along with popular navigation methods.

Adaptive Diffusion Terrain Generator for Autonomous Uneven Terrain Navigation
Youwei Yu" Junhong Xut Lantao Liu
Indiana University. Bloomington {youwyu, xul4, lantao]@iu.ed.
Abstract: Model-free reinforcement learning has emerged as a powerful method for developing robust robot control policies capable of navigating through com- plex and unstructured terrains. The effectiveness of these methods hinges on two essential elements: (1) the use of massively parallel physics simulations to expedite policy training, and (2) an environment generator tasked with crafting sufficiently challenging yet attainable terrains to facilitate continuous policy im- provement, Existing methods of environment generation often rely on heuristics constrained by of parameters, limiting the diversity and realism. In this work, we introduce the Adaptive Diffusion Terrain Generator (ADTG), novel method that leverages Denoising Diffusion Probabilistic Models to dynamically expand existing training environments by adding more diverse and complex terrains adap- tive to the current policy. ADTG guides the diffusion model's generation process through initial noise optimization, blending noise-corrupted terrains from existing training environments weighted by the policy's performance each correspond- ing enviromment. By manipulating the noise corruption level, ADTG seamlessly transitions between generating similar terrains for policy fine-tuning and novel ones to expand training diversity. Our experiments show that the policy trained by ADTG outperforms both procedural generated and natural environments, along with popular navigation methods.
Keywords: Curriculum Reinforcement Learning, Guided Diffusion Model, Field Robots
Introduction
Autonomous navigation across uneven terrains necessitates the development of control policies that exhibit both robustness and smooth interactions within challenging environments [1, 2, 3]. this work, we specifically target the training of a control policy that allows the mobile-wheeled robots adeptly navigate through diverse uneven terrains, such off-road environments characterized by varying elevations, irregular surfaces, and obstacles.
Recent advancements in reinforcement learning (RL) have shown promise in enhancing autonomous robot navigation on uneven terrains [4, 5, 6]. While an ideal scenario involves training an RL policy to operate seamlessly in all possible environments, the complexity real-world scenarios makes impractical to enumerate the entire spectrum of possibilities. Popular methods, including curricu- lum learning in simulation [7] and imitation learning using real-world collected data [8], encounter limitations in terms of training data diversity and the human efforts required. Without sufficient data and training, the application of learned policies to dissimilar scenarios becomes challenging, thereby hindering efforts to bridge the train-to-real gap. Additionally, existing solutions, such as scalar traversability classification for motion sampling [9, 10] and optimization methods [11, 12), may exhibit fragility due to sensor noise and complex characteristics of vehicle-terrain interactions.
Equal contribution. Demonstrations at https: http::/:s///aagiiiinooooooe github.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

#2:01
0.035
0.030
K-1000
K-750
No
0.020
0.6
0.4
K-500
W2
0.010
02
1.0
1000
Forward Steps
(a) Adjusting Realistic Terrain Diversity and Difficulty with ADTG
(b) Diffusion Variance
Figure I: (a) Each row shows denoised terrains at different forward steps K, and cach column blends terrains of varying difficulty using weighting factor u. Decreasing wi (easier terrain) and increasing w2 raises difficulty. As K increases, terrains show more novelty while maintaining difficulty. (b) Variance of typical denoising diffusion process with zoom-in view (left).
To tackle this challenge, we propose the Adaptive Diffusion Terrain Generator (ADTG), an environ- ment generator designed to co-evolve with the policy, producing new environments that effectively push the boundaries of the policy's 's capabilities. Starting with an initial environment dataset, which may be from existing elevation data terrains generated by generative models, ADTG is capable of expanding it into new and diverse environments. The significant contributions include:
1. Adjustable Generation Difficulty: ADTG dynamically modulates the complexity of generated terrains by optimizing the initial noise (latent variable) of the diffusion model. It blends noise- corrupted terrains from the training environments, guided by weights derived from the current policy's s performance. As a result, the reverse diffusion process, starting at the optimized initial noise, can synthesize terrains that offer the right level of challenge tailored the policy's s current capabilities.
2. Adjustable Generation Diversity: By adjusting the initial noise level before executing the DDPM reverse process, ADTG effectively varies between generating challenging terrains and introducing new terrain geometries. This capability is tailored according to the diversity present in the existing training dataset, enriching training environments as needed throughout the training process. Such diversity is crucial to ensure the trained policy to adapt and perform well in range of previously unseen scenarios.
We systematically validate the proposed ADTG framework by comparing it with established envi- ronment generation methods [13, 5] for training navigation policies on uneven terrains. Our exper- imental results indicate that ADTG offers enhanced generalization capabilities and faster conver- gence. Building on this core algorithm, we integrate ADTG with teacher-student distillation [14] and domain randomization [15] in physics and perception. We evaluate the deployment policy with zero-shot transfer to simulation and real-worls experiments. The results reveal our framework's superiority over competing methods [16, 10, 4, 17] in key performance metrics.
Related Work
Offroad Navigation. Off-road navigation requires planners to handle more than simple planar motions. Simulating full terra-dynamics for complex, deformable surfaces like sand, mud, and snow computationally intensive. Consequently, most model-based planners use simplified kinematics models for planning over uneven terrains [3. 18, 19, 20, 21] and incorporate semantic cost maps to evaluate traversability not accounted in the simplified model [2, 22]. Imitation learning (IL) methods [8, 23, 24] bypass terrain modeling by learning from expert demonstrations but require labor-intensive data collection. On the other hand, model-free RL does not require expert data and has shown impressive results enabling wheeled [7, 25, 4, 26] and legged robots [13, 5, 27, 28] traversing uneven terrains by training policies over diverse terrain geometries. However, the challenge is to generate realistic environments to bridge the sim-to-real gap. The commonly-used procedural generation methods [5, 13] are limited by parameterization and may not accurately reflect real-world terrain geometries. Our work addresses this by guiding diffusion model trained on natural terrains to generate suitable terrain surfaces for training RL policies.

---

optimization method Automatic Curriculum Learning and Environment Generation. Our method is form of au- tomatic curriculum learning [29, 30], where it constructs increasingly challenging environments to train RL policies. While one primary goal curriculum learning in RL is to expedite training efficiency [31, 32, 33], recent work shows that such automatic curriculum can a by-product of unsupervised environment design (UED) [34, 35, 36, 37, 38]. It aims to co-evolve the policy and an environment generator during training achieve zero-shot transfer during deployment. Unlike prior works in UED, the environments generated by our method are grounded in realistic envi- ronment distribution learned by diffusion model and guided by policy performance. Recently, concurrent work proposes Grounded Curriculum Learning [39]. It uses a variational auto-encoder (VAE) to learn realistic tasks and co-evolve a parameterized teacher policy to control VAB-generated tasks using UED-style training- In contrast, our work uses sampling- to control the diffusion model's initial noise for guided generation.
Controllable Generation with Diffusion Models. Controllable generation aims to guide pre- trained diffusion model to generate samples that are not only realistic but also satisfy specific criteria. commonly used strategy is adding guided perturbations to modify the generation process a pre-trained diffusion model using scores from the conditional diffusion [40, 41] or gradients of cost functions [42]. Another approach is to directly optimize weights of a pre-trained diffusion model so that the generated samples optimize some objective function. By treating the diffusion generation process Markov Decision Process, model-free reinforcement learning has been used to fine- tune the weights of pre-trained diffusion model [43, 44]. This approach can also be viewed as sampling from an un-normalized distribution, given a pre-trained diffusion model as a prior [45]. Our work is closely related to initial noise optimization techniques for guiding diffusion models [46, 47, 48]. Instead of refining the diffusion model directly, these methods focus on optimizing the initial noise input. By freezing the pre-trained diffusion model, we ensure that the generated samples remain consistent with the original data distribution. In contrast to existing approaches focusing on content generation, our work integrates reinforcement learning (RL) with guided diffusion train generalizable robotic policies.
3 Preliminaries
3.1 Problem Formulation
We represent the terrain using grid-based elevation map, denoted as e E RR where W and H represent the width and height, respectively. This terrain representation is widely adopted in motion planning across uneven surfaces. Similar to most works in training RL policies for rough terrain navigation [13, 5], we use existing high-performance physics simulators [49] to model the state transitions of the robot moving on uneven terrains 8t+1 p($t41/st, e). Here, S and a E A represent the robot's state and action, and each realization of the elevation (i.e., terrain) specifies a unique environment. An optimal policy π (aals e) can be found by maximizing the expected cumulative discounted reward. Formally,
T R an) -==0
8" = arg max (a soopsss)) emple) pappppppppppa (
(1)
where p(sa) is the initial state distribution and p(e) denotes the distribution over the environments. Due to the elevation e imposing constraints on the robot's movement, the policy optimized through Eq. (1) is inherently capable of avoiding hazards elevated terrains. We aim dynamically evolve the environment distribution p(e) based on the policy's 's performance, ensuring training efficiency and generating realistic terrain elevations. While constructing realistic state transition p(s++1 (8t, 0z, e) is also important for reducing the sim-to-real gap, we leave it to our future work.
3.2 Adaptive Curriculum Learning for Terrain-Aware Policy Optimization
A theoretically correct but impractical solution to E1.(1) is to train all possible terrains A (el e.,,..e.,,, with p(e) uniform distribution over A. However, the vast variability of terrain
3

---

geometries makes this unfeasible. Even if possible, it might produce excessively challenging or overly simple terrains, risking the learned policy to have poor performance [50]. Adaptive Curricu- lum Reinforcement Learning (ACRL) addresses these issues by dynamically updating the training dataset [51]. ACRL generates and selects environments that yield the most policy improvement. In our work, designing an effective environment generator is crucial. It should (1) generate realis- tic environments matching real-world distributions and (2) adequately challenge the current policy. Common approaches include using adjustable pre-defined terrain types [13], which offers control but may lack diversity, and generative models [52], which excel in realism but may struggle with precise policy-tailored challenges, In the following, we detail the proposed ADTG, which balances realism and sailooe terrain generation.
Adaptive Diffusion Terrain Generator
This section introduces the Adaptive Diffusion Terrain Generator (ADTG), novel ACRL generator that manipulates the DDPM process based on current policy performance and dataset diversity. We begin by interpolating between "easy" and "difficult" terrains in the DDPM latent space to generate terrains that optimize policy training. Next, we modulate the initial noise input based on the training dataset's variance to enrich terrain diversity, fostering broader experiences and improving the policy's generalization across unseen terrains. We use 6, co, and eş to denote the environment in the training dataset, the generated terrain through DDPM, and the DDPM's latent variable at timestep respectively. All three variables are the same size IR W Since in DDPM, noises and latent variables are the same [53], use them interchangeably.
4.1 Performance-Guided Generation via DDPM
Latent Variable Synthesis for Controllable Generation. Once trained, DDPM can control sample generation by manipulating intermediate latent variables. In our context, the goal is to steer the generated terrain surface maximize policy improvement after being trained it. While there are numerous methods to guide the diffusion model [40, 44], we choose to optimize the starting noise to control the final target [46]. This approach is both simple and effective, as it eliminates the need for perturbations across all reverse diffusion steps, as required in classifier-free guidance [40], or fine-tuning of diffusion models [44]. Nevertheless, still enhances the probability of sampling informative terrains tailored the current policy.
Consider subset of terrain elevations A (e) ee,,,?,..., e.) from the dataset A. To find initial noise that generates terrain maximizing the policy improvement, we first generate intermediate latent variables (noises) for each training environment in A at a forward diffusion time step k, of (( 1ee,,, k)) for = 1, 1,...,t, n. Assume that we have a weighting function w(e, x) that evaluates the performance improvement after training on each terrain map e We propose to find the optimized initial noise as a weighted interpolation of these latents, where the contribution of each latent ef. is given by the policy improvement in the original terrain environment w(e", TT)
c = -> w(e', π ))e()/ / """", mT]].
(2)
The fused latent variable e is then processed through reverse diffusion, starting at time & to synthe- size a new terrain e6. The resulting terrain blends the high-level characteristics captured by the latent features of original terrains, proportionally influenced by their weights. We illustrate this blending effect in Fig. 1. By controlling weights wi, we can steer the difficulty of the synthesized terrain.
Weighting Function. Policy training requires dynamic weight assignment based on current policy performance, We define the following weighting function that penalizes terrains that are too easy or too difficult for the policy:
w (e, m) exp ()),,))), r(e,T) (s(e, -5)2/02.
It penalizes the deviation of terrain difficulty, s(e, T), experienced by the policy from desired difficulty level 5. This desired level indicates terrain difficulty that promotes the most significant

---

improvement in the policy. The temperature parameter controls the sensitivity of the weighting function to deviations from this desired difficulty level. We use the navigation success rate [54] to represent .(:.)) While alternatives like TD-error [55] or regret [56] exist, this metric has proven to be an effective and computationally efficient indicator for quantifying an environment's potential to enhance policy performance in navigation and locomotion tasks [5, 13J. In Appendix A, we show that the optimized noise in Eq. (2) and the corresponding weighting function in Eq. (3) can be derived from formulating the noise optimization problem using Control as Inference [57, 58] and solving it through Importance Sampling. We denote the procedure of optimizing the noise ep using Eq. 2 and generating the final optimized environment by reverse diffusion starting at E as = Synthesize (A, æ, k), where l is the starting time step of the reverse process. As discussed the next section, a large l crucial to maintaining diversity.
4.2 Diversifying Training Dataset via Modulating Initial Noise
The preceding section describes how policy performance guides DDPM in generating terrains that challenge the current policy's ' capabilities. As training progresses, the pool challenging terrains diminishes, leading to point where each terrain no longer provides significant improvement for the policy. Simply fusing these less challenging terrains does not create more complex scenarios. Without enhancing terrain diversity, the potential for policy improvement plateaus. To overcome this, it is essential to shift the focus of terrain generation towards increasing diversity. DDPM's reverse process generally starts from a pre-defined forward step, where the latent variable is usually pure Gaussian noise. However, it can also start from any forward step K with sampled noise as ek ~ q(ex]es (ea) [59]. Fig. I shows the variance of generated terrains decreases with fewer forward steps and vice versa. To enrich our training dataset's 's diversity, we propose the following:
1. Variability Assessment: Compute the dataset's variability Avar by analyzing the variance of the first few principal components from Principal Component Analysis (PCA) on each elevation map. This serves an efficient proxy for variability.
Forward Step Selection: The forward step k λ A is inversely proportional the variance. We use a linear scheduler: k K(1 Avar), with K the maximum forward step and Avar normalized to 0 ~ 1. This inverse relationship ensures greater diversity in generated terrains. Terrain Generation: Using the selected forward step k; apply our proposed Synthes ze to generate new terrains, thus expanding variability for training environments.
Algorithm 1 ACRL with Adaptive Diffusion Ter- rain Generator
4.3 ACRL with ADTG
We present the final method pseudo-coded in Alg. 1 using the proposed ADTG for training privileged policy. The algorithm iterates over policy optimization and guided terrain genera- tion, co-evolving the policy and terrain dataset until convergence. The algorithm starts by se- lecting a training environment that provides the best training signal for the current policy, which can be done in various ways [50]. For example, one can compute scores for terrains based on the weighting function in Eq. (3) and choose the one with the maximum weight. Instead of choosing deterministically, we sample the terrain based on their corresponding weights. The Opt im step collects trajectories and performs one policy up- date in the selected terrain. After the update, we evolve the current dataset by generating new one, as shown in lines 4 of Alg. 1. In practice, we run Alg. in parallel across N terrains, each with multiple robots. In parallel training, Synthesi ize begins by sampling N x n initial noises, where N is the number of new terrains (equal the number of parallel environments) and is the sample size in Eq. (2). It then optimizes over these noises to generate N optimized noises. Finally, these optimized noises are passed to the DDPM to generate N terrains. When the dataset grows large,
Input: Pretrained DDPM ((-,;g)) an initial ter- rain dataset
Output: The optimized privileged policy x² Initialize: The privileged policy π
1: while x not converge do
2:
Se Selector(A, f Optim(T,
> Env. Selection Policy Update Seec 4.2
3:
4:
= ([1-Avar))
5:
é' Synthesize(A.. k) Sec. 4.1
6:
A +- AU
b Update Dataset
7: end while
5

---

it sub-samples terrains from Selector's complement, with success rates updated by the current policy. We validate effectiveness of Se lector in Sec. 5.1 and Synthesize ze in Appendix A.3, which also explains its sub-sampling logic.
ADTG with Policy Distillation. While ADTG can train policies generalize over terrain ge- ometries, real-world deployments face challenges beyond geometry. This includes noisy, partial observations and varying physical properties. To address these, we distill privileged policy,
which observes the complete elevation map and noiseless states into a depth vision-based policy with noisy measurements. The privileged policy is trained using PPO [60], and the deployment policy trained using DAgger [6]], utilizing methods sim- ilar to [6]. To enhance generalization,
AOTG
8DPM
(PPO)
Teacher
Ducoch Minta
Encorder
CNN
MLP
Loss
integrate physical and perception domain randomization [15] (more details in Appendix. C). The training loop in-
CNN
+
Losont
Action
Polccegedt State
Hocky
tegrating ADTG with teacher-student distillation illustrated in Fig. and the Appendix. With ADTG validated Sec. 5.1, our whole system demon- strated in the sim-to-deploy experiments.
CNN
Zero-Shee Sim-to-Real
(DAgger)
DEPLOYMENT
Figure 2: Framework with our Adaptive Diffu- sion Terrain Generation (ADTG) and Policy Dis- tillation. Model-free RL trains privileged policy on ADTG-generated terrains. The privileged pol- icy is then distilled into the deployment (Learner) policy using data aggregation. Iterative training and terrain generation through ADTG enhance the deployment policy's generalization.
Experiments
In this section, we start with the algorithmic eval- uation to highlight the effectiveness of our ADTG. Then through sim-to-sim on two kinds of wheeled robot platforms and sim-to-real on wheeled and one quadruped robot, we study the performance of our ADTG policy against ablations and competing methods.
We train in IsaacGym [49] and parallel 100 uneven terrains, each with 100 robots. Among the elevation dataset [62] as detailed in Appendix C.I, we select 3000 for DDPM training (E-3K), 100 for algorithmic evaluation (E-IH), and 30 for sim-to-sim experiments (E-30). Simulations run on an Intel 19- 14900KF CPU and NVIDIA RTX 4090 GPU. Real-world tests use NVIDIA Jetson Orin.
5.1 Algorithmic Performance Evaluation
Normeted Betum
This section evaluates whether the environment cur- riculum generated by ADTG enhances the gener- alization capability of the trained privileged policy across unfamiliar terrain geometries, on the wheeled ClearPath Jackal robot. We compare with the fol- lowing baselines. Procedural Generation Curriculum (PGC), commonly used method, uses heuristically designed terrain parameters [13]. Our implemented PGC follows ADTG, adapting the terrain via the score function Eq. (3) and dynamically updating the dataset. First, to ablate our Adaptive curriculum, Diffusion Terrain Generator (ADTG) generates terrain without curriculum, Procedural Generation (PGC) randomly samples parameters. To ablate our Diffusion Generator, Natural Adaptive Terrain (N-ADTG) selects terrains directly from H-3K. To ablate both, Natural Terrain (N-ADTG) randomly samples from E-3K without curriculum. Mono font means the ablated parts. Second, to test ADTG's ' robustness to initialization other than diffusion, N-ADTG and P-ADTG start with datasets sub-sampled from E-3K and procedural generations.
Epach
Figure 3: The comparison of the normalized re- turn among proposed ADTG, the haselines, and ablation methods.
All methods use the same training and evaluation setup. After each training epoch, policies are tested in held-out evaluation environments with 200 000 start-goal pairs. Fig. shows the normalized RL return'. ADTG outperforms PGC and N-AT within 40 epochs, exceeding the return by over 0.1,
It is calculated by the actual return divided by value function's upper bound based reward function.

---

Suc.
Rate (%)
Traj. Ratio
Orien.
Vib. (ndd)
Orien. Jerk
Pos. Jerk (
S
D
R
D
S
R
D
D
Falco
26
47
22
2.76
1.24
1.18
0.71
0.15
0.17
275.6
165
143.2
48
42.1
20.3
MPPI
48
71
34
1.21
1.23
1.04
0.75
0.18
0.18
228.7
161.7
81
40.7
34.2
23.6
TERP
33
25
1.62
1.16
1.1
0.77
0.13
0.13
210.1
137.6
112.2
37.1
20.2
17.5
POVN
17
28
24
1.23
1.23
1.28
0.68
0.18
0.14
241
120.2
127.5
43.7
38.6
21.3
N-AT
67
X
1.24
1.08
323.4
X
X
57.6
PGC
43
1.92
0.97
236.1
42
Ours
87
45
1.52
1.25
1.32
0.65
0.11
0.13
193.5
113.3
73.4
20
18.1
Table Statistical results for S(imulation), woral-world) and D(une) comparing our method with SOTA works. S: 30 environments, R: grass, forest, arid, rust, mud, and gravel; D: dune hard, tough, and expert. Green and Bold indicate the best and second- best results. x means poor performance.
which translates to more than 20 000 successes in our recorded data. As shown in N-ADTG and P-ADTG, regardless of initial terrains, ADTG consistently generates effective terrains. DTG, N-T, and PG results validate our curriculum Selector, with PG's sharp curve changes due to overly difficult terrains. N-AT lacks dataset evolution, and PGC lacks efficient terrain parameter control. In summary, ADTG excels at adapting environment difficulty based on evolving policy performance.
5.2 Zero-shot Sim-to-Sim and Sim-to-Real Experiments
This section evaluates the zero-shot transfer capability of sim-to-deploy environments. Metrics in- clude the success rate, trajectory ratio, orientation vibration [w), orientation jerk a |. and position jerk dt where w and denote angular velocity and linear acceleration. These motion stability in- dicators are crucial in mitigating sudden pose changes and enhancing overall safety. The trajectory ratio is the successful path length relative to straight-line distance and indicates navigator efficiency. All baselines use the elevation map [63] with depth camera and identify terrains as obstacles if the slope estimated from the elevation map exceeds 20 For orientation costs, we obtain the robot's roll and pitch by projecting its base to the elevation map.
0e
Baselines. Falco [16]. a classic motion primitives planner, and MPPI [10, 64], sampling-base model predictive controller, are recognized for the success rate and efficiency. They use the point- cloud and elevation map to weigh collision risk and orientation penalty. TERP [4), an RI. policy trained in simulation, conditions on the elevation map, rewarding motion stability and penalizing steep slopes. POVNav [17] performs Pareto-optimal navigation by identifying sub-goals in seg- mented images [65], excelling in unstructured outdoor environments.
Simulation Experiment. We simulate wheeled robots, ClearPath Jackal and Husky, in ROS Gazebo on 30 diverse environments (E-30), equipped with a RealSense D435i camera. We add Gaussian noises to the robot state, depth measurement, and vehicle control to introduce uncertainty. 1000 start and goal pairs are sampled for each environment. We do not include ablations other than N-AT because of poor algorithmic performance. As Jackal's results shown in Table 1, our method outper- forms the baselines. Appendix C.2 provides statistical results for Husky. While all methods show improved performance due to the Husky's better navigability on uneven terrains, our method consis- tently outperformed baseline methods. The depth measurement noise poses a substantial challenge in accurately modeling obstacles and complex terrains. Falco and MPPI often cause the robot to get stuck or topple over, and TERP often predicts erratic waypoints that either violate safety on eleva- tion map or are overly conservative. Learning-base TERP and POVN lack generalizability. with their performance varying across different environments. This issue is mirrored in N-AT and PGC, highlighting the success of adaptive curriculum and realistic terrain generation properties of ADTG.
The Jackal robot is equipped with a Velodyne 16 LiDAR, a RealSense D435i camera, and a 3DM-GX5-22 IMU. We test in 9 diverse representative environments, grass, forest, arid, rust, mud, gravel, dune-hard, rd, dune-tough, and dune cane-expert, dis- played in Fig. 4. For each environment, we sample 36 start-goal pairs. Note that the results of N-AT and PGC policies are not presented they achieved limited success in real-world experiments. Table I provides an overview of the performance metrics, with success rates across all methods de-
Real-world Experiment.
7

---

Dpan
Start Goal
Waypolinss
Camera
Ours
MPPI
TERP
daper
(b) Robot and Dune-HHad Example Demonstrations Figure 4: The left panel shows nine challenging environments, the middle our platform, and the right Dune- hard, where method outperformed others navigating ravines.
(a) Experimental Environments
tailed in the Appendix C.3. In forest, POVNav struggled with image segmentation of complex ground elements like foliage and underbrush. Other baselines encountered difficulties due to impre- cise depth data affecting terrain modeling. All baselines risked getting stuck in muddy, sandy, and rocky areas. In contrast, our method autonomously engaged in are movements, avoiding immobi- lization in slippery environments. In dunes, characterized by steep elevation changes, failures of other methods were due to incorrect obstacle detection or severe slope changes causing overturn.
In addition, we ablated Physics and Perception domain randomization to study their contributions. Removing either reduced success rates, though both ablations outperformed baselines due to ADTG. The perception domain suffered from camera depth issues tied to exposure. Without the physics do- main, the robot showed improved orientation smoothness but struggled to make reactive behaviors, since the learned arc movements in slippery areas benefited from this domain.
Real-Worle World Quadruped Locomotion. To validate ADTG's generalization across different embodi- ments, we benchmark against PGC and the built-in MPC controller the Unitree Gol 1 quadruped in the same environments as Jackal. Both ADTG and PGC were trained using the Parkour [66] "walk" policy. Similar to the hiking experiment [5, 13], we conducted continuous 1.2 km loop, following MPC's footprints for fairness. Faitures, defined as rollovers, occurred when detours around challeng- ing areas were not allowed. MPC had 6 rollovers 3 from high-speed commands, on challenging gravel and dune terrains). ADTG policy exhibited natural postures with fallures due to high-speet commands. PGC policy showed robust prostrate postures but 11 failures due sudden movements (jumps) all terrains except grass. The results highlight ADTG's advantages in learning robust behavior. See the appendix for the full Jackal ablation analysis and quadruped experiment details.
6 Conclusion, Limitations and Future Directions
We propose an Adaptive Diffusion Terrain Generator (ADTG) to create realistic and diverse terrains based on evolving policy performance, enhancing RL policy's generalization and learning efficiency. To guide the diffusion model generation process, we propose optimizing the initial noises based on the potential improvements of the policy after being trained on the environment denoised from this initial noise. Algorithmic performance shows ADTG's superiority in generating challenging but suitable environments established methods such commonly used procedural generation curriculum. Combined with domain randomization in teacher-student framework, it trains a robust deployment policy for zero-shot transfer to new, unseen terrains. Extensive -e-ploy tests with wheeled and quadruped robots validate our approach against SOTA planning methods.
Limitations and Future Directions: A key limitation of ADTG that evolves only the envi- ronment distribution, relying on physics simulators for state transitions, limiting deployment in the complex real world. While domain randomization helps, it's not full solution. Future work will integrate environment distribution and physics to bridge the sim-to-real gap. Additionally, ADTG's environment scale is suited for local planning, but larger environments are needed for long-horizon tasks. We plan to explore hierarchical diffusion models for generating multi-layere environments.

---

References
[1] Z. Jian, Z. Lu, X. Zhou, B. Lan, A. Xiao, X. Wang, and B. Liang. Putn: plane-fittinn based uneven terrain navigation framework. JEEE/RSJ Inl. Conf. on Intel. Robots and Syst. (IROS), pages 7160-7166. IEEE, 2022.
[2] X. Meng, N. Hatch, A. Lambert, A. Li, N. Wagener, M. Schmittle, J. Lee, W. Yuan, Z. Chen, S. Deng, et al. TerrainNet: Visual Modeling of Complex Terrain for High-speed, Off-road Navigation. Robotics:
Science and Systems, 2023.
[3] Xu, K. Yin, Z. Chen, J. M. Gregory, A. Stump, and L. Liu. Kemel-based diffusion ap- proximated Markov decision processes for autonomous navigation and control on unstructured terrains. The International Journal of Robotics Research, page 0278364923 122597777 2024.
[4] K. Weerakoon, A. J. Sathyamorrthy, U. Patel, and D. Manocha. TERP: Reliable Planning in Uneven Outdoor Environments using Deep Reinforcement Learning. Proc. Int. Conf. Robot. Automat., pages 9447-9453, 2022.
[5] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust percep- tive locomotion for quadrupedal robots in the wild. Science Robotics, 7(1 xeaab22222 2022.
[6] A. Agarwal, Kumar, Malik, and D. Pathak. Legged Locomotion in Challenging Terrains using Egocentric Vision. In Proc. Conf. Robot Learn., number 2, pages 403-415, 2023.
[7] S. Josef and A. Degani. Deep Reinforcement Learning for Safe Local Planning of a Ground Vehicle in Unknown Rough Terrain. IEEE Robot. and Automat. Letters, 5(4):6778-6755, 2020.
[8] X. Xiao, Biswas, and Stone. Learning Inverse Kinodynamics for Accurate High-Speed Off-Road Navigation on Unstructured Terrain. IEEE Robot. and Automat. Letters, 6(3):6054- 6060, 2021.
[9] D. Fox, W. Burgard, and S. Thrun. The dynamic window approach to collision avoidance. IEEE Robot, & Automat. Magazine, 4(1):23-33, 1997.
[10] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou. Aggressive driving with model predictive path integral control. In Proc. IEEE Int. Conf. Robot, Automat. pages 1433-1440, 2016.
[11] C. Rösmann, Hoffmann, and T. Bertram. Timee-Elastic-Baan for time-optimal point-to- point nonlinear model predictive control. In European Control Conf., pages 3352-3357, 2015.
[12] X. Zhang, A. Liniger, and F. Borrelli. Optimization-Base Collision Avoidance. IEEE Trans. on Control Sys. Tech., 29(3)9972-983, 2021.
[13] Lee, Hwangbo, L. Wellhausen, Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science Robotics, 5(4))ceabc5986, 2020.
[14] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl. Learning by Cheating. In Proc. of the Conf. on Robot Learning, volume 100, pages 66-75. PMLR, 30 Oct-01 Nov 2020.
[15] I. Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, al. Solving rubik's cube with robot hand. arXiv preprint 1rX1v:1910007113. 2019.
[16] J. Zhang, C. Hu, R. G. Chadha, and S. Singh. Falco: Fast likelihood based collision avoidance with extension to human -omnan- navigation. Journal of Field Robot., 37:1300 - 1313, 2020.
[17] D. Pushp, Z. Chen, C. Luo, J. M. Gregory, and L. Liu. POVNav: A Pareto-Optimal Mapless Visual Navigator, 2023.

---

[18] J. Wang, L. Xu, H. Fu, Z. Meng, C. Xu, Y. Cao, X. Lyu, and F. Gao. Towards Efficient Trajectory Generation for Ground Robots beyond 2D Environment. In Proc. IEEE Int. Conf. Robot. Automat. pages 7858-7864, 2023.
[19] I. Xu, K. Chai, Z. Han, H. Liu, C. Xu, Y. Cao, and F. Gao. An Efficient Trajectory Planner for Car-Like Robots on Uneven Terrain. In IEEE/RSJ Int. Conf. on Intel. Robots and Syst. (IROS), pages 2853-2860. IEEE, 2023.
[20] H. Lee, J. Kwon, and C. Kwon. Learning- Uncertainty-awarr Navigation in 3D Off-Road Terrains. In Proc. Ini. Conf. Robot. Automat. pages 10061-10068, 2023.
[21] Moyalan, Y. Chen, and U. Vaidya. Convex Approach to Data-Driven Off-Road Navigation via Linear Transfer Operators. IEEE Robot. and Automat. Letters, 8(6):3278-3285 2023.
[22] Triest, M. G. Castro, Maheshwari, M. Sivaprakasam, Wang, and S. Scherer. Learning Risk-Aware Costmaps via Inverse Reinforcement Learning for Off-Road Navigation. Int. Conf. on Robot, and Automat., pages 924-930, 2023.
[23] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. A. Theodorou, and B. Boots. Imitation learning for agile autonomous driving. The International Journal of Robotics Research, 39 (2-3):286-302 20288660222100 2020.
[24] S. Siva, M. Wigness, J. Rogers, and H. Zhang. Enhancing Consistent Ground Maneuverability by Robot Adaptation to Complex Off-Road Terrains. In Conf. on Robot Learn., 2021.
[25] H. Hu, K. Zhang, A. H. Tan, M. Ruan, C. Agia, and G. Nejat. A Simlto-Real Pipeline for Deep Reinforcement Learning for Autonomous Robot Navigation in Cluttered Rough Terrain. IEEE Robot. and Automat. Letters, 6(4):6569-653 6576, 2021.
[26] U. Patel, N. K. S. Kumar, A. J. Sathyamootthy, and D. Manocha. DWA-RL: Dynamically Feasible Deep Reinforcement Learning Policy for Robot Navigation among Mobile Obstacles. In IEEE Inf. Conf. on Robot. and Automat. (ICRA), pages 6057-6063, 2021.
[27] F. Jenelten, J. He, F. Farshidian, and M. Hutter. Robotics, 9(6))eadh5401, 2024.
DTC: Deep Tracking Control. Science
[28] Miki, Lee, Wellhausen, and M. Hutter. Learning to walk in confined spaces using 3D representation. arXiv preprint arXiv:2403.001 001440 2024.
[29] R. Portelas, C. Colas, L. Weng. K. Hofmann, and P.-Y. Oudeyer. Automatic curriculum learn- ing for deep rl: A short survey. arXiv preprint arXiv:2003. 046644 2020.
[30] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey. Mach. Learn. Res., 21(1), 2020. ISSN 1532-4435.
[31] D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver. Distributed Prioritized Experience Replay. In Int. Conf. on Learn. Representations, 2018.
[32] M. Fang, T. Zhou, Y. Du, L. Han, and Z. Zhang. Curriculum-guide Hindsight Experience Replay. In Advances in Neural Info. Processing Syst. volume 32. Curran Associates, Inc., 2019.
[33] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic Goal Generation for Reinforcement Learning Agents. In Proc. of Int. Conf. Mach. Learn, volume 80, pages 1515-1528. PMLR, 10-15 Jul 2018.
[34] M. Dennis, N. Jaques, Vinitsky, A. Bayen, Russell, Critch, and Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in Neural Info. Processing Syst., 13:13049-1306, 2020.
10

---

[35] R. Wang, J. Lehman, J. Clune, and K. O. Stanley. Paired open- trailblazer (poet): End- lessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint 1eXXvv:901.0111 2019.
[36] R. Wang, Lehman, Rawal, J. Zhi, Y. Li, Clune, and Stanley. Enhanced POET: Open-Ended Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions. In Int. Conf. Mach. Learn., 2020.
[37] M. Jiang, E. Grefenstette, and T. Rocktäschel. Prioritized level replay. In Int. Conf. on Mach. Learn., pages
PMLR, 2021.
[38] D. Li, W. Li, and P. Varakantham. Diversity Induced Environment Design via Self-Play, 2023.
[39] L. Wang, Z. Xu, P. Stone, and X. Xiao. Grounded Curriculum Learning. arXiv preprint arXiv:2409. 19816, 2024.
[40] Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598: 2022.
[41] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision makinn? arXiv preprint arXiv:2211.15657 2022.
[42] Z. Zhong, D. Rempe, D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone. Guided conditional diffusion for controllable traffic simulation. IEEE Int. Conf. on Robot. and Automat. (ICRA), pages 3560-3566, IEEE, 2023.
[43] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
[44] M. Uehara, Y. Zhao, T. Biancalani, and S. Levine. Understanding Reinforcement Boaming-Based Fine- Tuning of Diffusion Models: A Tutorial and Review. arXiv preprint 24XY::2407.13334 2024.
[45] S. Venkatraman, M. Jain, L. Scimeca, M. Kim, M. Sendera, M. Hasan, L. Rowe, S. Mittal, Lemos, E. Bengio, al. Amortizing intractable inference in diffusion models for vision, language, and control, arXiv preprint arXiv:2405.20971 2024.
[46] H. Ben- HemmH O. Puny, I. Gat, B. Karrer, U. Singer, and Y. Lipman. D-Flow: Differentiating through Flows for Controlled Generation. arXiv preprint vrXiv:2402.14017 2024.
[47] K. Karunratanakul, K. Preechakul, E. Aksan, T. Beeler, S. Suwajanakorn, and S. Tang. Opti- mizing diffusion noise can serve as universal motion priors. In JEEE/CVF Conf. on Computer Vision and Pattern Recognition, pages
2024.
[48] X. Guo, Liu, M. Cui, J. Li, H. Yang, and D. Huang. Initno: Boosting text-to-image diffusion models via initial noise optimization. In JEEE/CVF Conf. on Computer Vision and Pattern Recognition, pages 9380-9389, 2024.
[49] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[50] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. Int. Conf. on Mach. Learn., pages 41-48, 2009.
[51] R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y. Oudeyer. Automatic curriculum leamn ing for deep RL: A short survey. arXiv preprint arXhv:2003.04664, 2020.

---

[52] A. Jain, A. Sharma, and Rajan. Adaptive & Multi-Resolution Procedural Infinite Terrain Gen- eration with Diffusion Models and Perlin Noise. In Proc. of the Thirteenth Indian Conference on Computer Vision Graphics and Image Processing, 2023. ISBN 9781450398220.
[53] J. Ho, A. Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Info. Processing Syst., volume 33, pages 9840-6851 Curran Associates, Inc., 2020.
[54] C. Florensa, D. Held, X. Geng, and Abbeel. Automatic Goal Generation for Reinforce- Learning Agents. In Ini. Conf. on Mach. Learn., volume 80 of Proceedings of Machine Learning Research, pages 1515-1528. PMLR, 10-15 Jul 2018.
[55] M. Jiang, M. Dennis, J. Parker-Holder, J. Foerster, E, Grefenstette, and T. Rocktäschel. Replay- guided adversarial environment design. Advances Neural Info. Processing Syst., 34:1884- 1897, 2021.
[56] Parker-Holder, M. Jiang, M. Dennis, M. Samvelyan, J. Foerster, Grefenstette, and Rocktäschel. Evolving curricula with regret-based environment design. In Int. Conf. on Mach. Learn., pages 17473-17498. PMLR, 2022.
[57] H. J. Kappen, V. Gómez, and M. Opper. Optimal control as a graphical model inference problem. Machine learning, 87: 87:159-182 2012.
[58] S. Levine. Reinforcement learning and control probabilisstic inference: Tutorial and review. arXiv preprint arXiv:1805.000 2018.
[59] C. Meng, He, Song, J. Song. J. Wu, J. Zhu, and S. Ermon. Sdedit: Guided image syn- thesis and editing with stochastic differential equations. Int. Conf. on Learn. Representation, 2022.
[60] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Al Algorithms, 2017.
[61] S. Ross, G. Gordon, and D. Bagnell. A Reduction of Imitation Learning and Structured Pre- diction to No-Regret Online Learning. In Proc. of Machine Learn. Research, volume 15, pages 627-635, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR.
[62] S. Krishnan, C. Crosby, V. Nandigam, M. Phan, C. Cowart, C. Baru, and R. Arrowsmith. OpenTopporaphy: a services oriented architecture for community access to LIDAR topogra- phy. In Proc. of the 2nd international conference on compuling for Geospatial Research & Applications, pages 1-8, 2011.
[63] P. Fankhauser, M. Bloesch, and M. Hutter. Probabilistic Terrain Mapping for Mobile Robots with Uncertain Localization. IEEE Robot. and Automat. Letters, 3(4):3019-3083 2018.
[64] Mohamed, K. Yin, and Liu. Autonomous Navigation of AGVs in Unknown Cluttered Environments: Log-MPPI Control Strategy. IEEE Robot, and Automat. Letters, 7(4):10240- 10247, 2022.
[65] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention Mask Transformer for Universal Image Segmentation. In IEEF/CVE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1280-1289, 2022.
[66] Z. Zhuang, Z. Fu, J. Wang, C. G. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot Parkour Learning. In Conf. on Robot Learn., 2023.
12

---

Appendix
A
Initial Noise Optimization by Control Inference
In this section, we justify our algorithm design through the lens of control-as-inference [1, 2] to optimize the initial noise of diffusion models.
A.1 Denoising Diffusion Probabilistic Models Preliminary
Denoising Diffusion Probabilistic Models (DDPMs) [5] are generative models that generate original data from Gaussian noise through series of forward and reverse steps. In the forward process, Gaussian noise is gradually added to the original terrain sample until final step K, with the terrain at each step & distributed as q (e& (e=-1) = N (exlv1 - 3, We use cosine noise schedules [6] (B,,... 3x) ). The reverse process gradually reduces the noise from e to generate the original data ep using noise predictor e(ex, k; Φ). The training loss function for the noise predictor (()) MSE( (Cg + <x. k; o)).
A.2 DDPM Initial Noise Optimization
Let plex-1les; 0) denote single reverse diffusion step, with 0 the diffusion model's parameters. Based on [3], we can formulate iterative denoising of the DDPM as a Markov Decision Process (MDP), where the state evolution (policy and transition function combined) is the reverse diffusion itself p(*k+1|sk ax) = p(ex-1lex; 8), and the reward function is unless at the initial step k = 0:
R(ck: x) [((eo,T) T) if =
(4)
otherwise
In our work, r(eo, T) evaluates the improvements of the current policy if trained on the generated environment cp- The return of the noise eg starting at timestep k: as
I (ex) E [r(en, T) T) p(e 0)]
(5)
where t AJ. Our goal is to find the noise that maximizes the Eq. (5), arg maxe, H(e)) Gradient-basee methods using differentiable rewards [4] are challenging to apply in our setup, as computing the gradient of policy improvement with respect to the initial noise is impractical. This is because evaluating policy improvement requires simulating the policy on each generated terrain to compute success rates. Instead, we frame this as a control-as-inferenn problem, using approximate sampling optimize the initial noise for Eq. (5). Following the KL control theory derivation [1], aim to find distribution of the starting noise ÿ(ex) that optimizes Eq. (5) while remaining close to reference noise distribution q(ex). This is achieved by adding extra KL cost to the return in Eq. (5) and taking expectation with respect to the random initial noise
J(ek) - log ((Ex) g(ex)
[(x)) E
6x ~ ((ex)
(6)
/ ((ex)
(I(ex) - log ((ex) des
g(ex))
(xx (T(e)))) log g(Ex) dek
(lex))
q(ex)
/ (((x)) log q(ex) exp (e())
((ex)
dex
KL (Q(e))ll())()))
where (ex) x g(ex) exp exp (ex) an unnormalized distribution with high density at the high return region. The Kullback-Leibler (KL) divergence is non-negative, implying that the optimal return is achieved when vb. Consequently, sampling from & is equivalent to sampling from the optimal
noise distribution. While there are multiple methods sample from v, we employ importance
13

---

sampling to compute the expected value of the noise. This approach is chosen due to its simple computational structure within this formulation. The importance sampling can then he expressed as
1 / exa(es) exp (ex) des
Z E lexlex ~ {b(e))]
1
E (exp (I(e=)} ex {(ex)]
Z
1
Σ exp (f(e))
7
==1
where the samples are drawn from the reference distribution. The normalization constant can be estimated similarly
= / ((ex) exp (I(ex) de:
(8)
E eexp (T(e)) le {(x)]]
exp (F(e)))
Combining Eq.(7) and (8), we obtain the expected optimal noise
E exp (I(e))
[eslee ~ (bxx))
E exp (I (ex))
We define the reference distribution as the marginal distribution of the forward diffusion process q(ex) f (pee),, where p(e) is the distribution over the current training environment dataset. Sampling from g(ex) involves drawing from p(c) and then from g(<xle), which is ex- actly the process described in Section 4.1. This process is efficient because the forward diffusion q(<xle) is easy to sample, and we can readily draw samples from p(e) using the available environ- ment dataset. The steps in our algorithm directly correspond to the computation of the final expected noise expressed Eq. (7). Table 2 illustrates the mapping between these algorithmic steps and the computational steps of computing Eq. (7).
Control as Inference
ADTG
Action/State space
Space of initial noise
Optimal policy
Optimal initial noise distribution
Optimal action
Optimal initial noise ch
Negative deviation from desired success rate - (s((e, .τ) s)²/o²
Reward
Sample from training dataset e ~ p(e) (Appendix A.3) Sample from forward diffusion process ~ g(ex
Sample from proposal ek q(ex)
Importance weight
Weighting function expr T) (Eq. (3))
Importance sampling solver (Eq. (7))
Weighted interpolation (Eq, (2))
Table 2: Correspondence of algorithmic steps in ADTG with control as inference using importance sampling.
A.3
Environment Difficulty Manipulated by Diffusion Synthesis
The core assumption underlying our method is that the success rates of the generated terrains by ADTG are consistent with the weighted combination the success rates selected terrains. This appendix provides empirical evidence to support this assumption. During training, we sample ter- rains with the success rate outside the 0.6 to 0.85 range for Synthes ze. our dataset expands rapidly, we uniformly sub-sample up 1000 terrains the current (sub-)dataset. If the synthe- sized terrain's 'ss predicted success rate falls outside the 0.6 to 0.85 range, we adjust by adding suit- able samples while ensuring the total number under 16. We choose 16 based on the GPU memory
14

---

limitation. We first set aside 100 terrains dataset DD and assess their success rates using the priv- ileged policy. For each synthesized terrain, Selector sub-samples N terrains from D, where N follows the curriculum. For each synthesized terrain, we define the predicted success rate as = -Ei= w,s:)/( wt), where w and denote Synthesiz weight and success rate, re- spectively, The validation involves comparing the actual success rates evaluated by the privileged policy against the predicted success rates. Fig. illustrates this comparison across 8 diffusion steps, with each step containing 60 synthesized samples. For clarity, each step's display is organized into two rows. The first row presents the predicted success rates, and the second row shows the actual success rates. The similarity in color between these two rows indicates the consistency of the suc- cess rates for the generated terrains. Our result shows that most of the samples closely align with the predicted success rates.
10 an Diffusion Step
Samp es
Figure 5: Synthesized terrains' success rates by diffusion model. For each diffusion step with 60 samples, the first presents predicted success rates, and the second shows actual success rates. This shows effective terrain difficulty manipulation by our Synthesia ie function.
B
System Design
B.1
Privileged Policy
The privileged (teacher) policy is trained using Proximal Policy Optimization (PPO) [7] for goal- oriented navigation on uneven terrains, minimizing motion vibration and jerk. The design as following:
State, Observation, and Action. The state s. = [ge, Ut, Wt, de, af-1 1 at timestamp t is ground-truth motion information, including the orientation in quaternion E IR, linear velocity D, E R angular velocity We E R3, relative goal distance de E IR?, and previous action af_1 E RR The reference frame is anchored to the robot base. Normalization applied to linear and angular velocities. We refrain from normalizing the relative goal distance, as this allows the policy adapt varying goal range scales. The observation has full access to the elevation map o" E 1128x 128 Each observation remains constant during an ACRI. episode when ADTG does not evolve terrains. The action a. = la,", awwif applied to the robot represents proportional-derivan forward linear and yaw angular velocities, where af = [u] wi] is the network output, and [o aw) are coefficients for the vehicle drive system.
(PD) targets for
of the weighted components: Goal Proximity e1 1, (115,(8) (st) assesses the robot's closeness to the goal against thresholds 6pcs and 8 Orientation Regulation 02 11s,, (F((:)) imposes penalties for exceeding safe roll and pitch angles, Movement Consistency cal calla, dt-1/12 + c4 callá, - /d-1/12 incentives consistent and gradual actions, Ground- contact and Safety c; dim(f = 0) + cs dim(f > 55) discourages situations where the contact force mean ground contact loss or a collision risk. The total reward is parameterized as follows:
Rewards. The total reward is structured
5.0 50.25 0.001 X 1 hm (St)
0.01 x 16,></12(4;)
(10)
0.01 x ||a 0.-1112 - 0.0001 x ||á - á
0.01 X dim(f 100) - 0.001 x dim(f < 0.00001).
15

---

Start and Termination. Episodes start with robots in random poses and assigned navigation targets. The initial state so is sampled by position, velocity, and yaw, with other parameters managed by the physics engine. The state 2 considers terrain geometry to avoid immediate failure. Initially, goals are within a circular sector, with a radius dmax = [0.5, 3.0Jm and central angle of |-x/3, n //33 rad. As the policy progresses success rate of 60%, the radius and angle incrementally increase by 0.5m and æ 1/18 until semicirele with radius of 6m. An episode ends upon reaching the goal or triggering safety running length constraints. The maximum episode length is d/aax/ AAl//AA//aa Al/Vavg with Havg = 0.3 m/s the minimum average velocity and A = 0.005 the simulator timestep. Early termination occurs if the roll or pitch angle exceeds ōqua т 7/99. or if the robot collides with the environment, or if two or more wheels are off the ground, risking toppling or entrapment.
B.2 Deployment Policy
The deployment (student) policy 0 trained via Dataset Aggregation (DAgger) [8] to minimize the mean squared error match the teacher's actions. At each timestamp the deployment policy observes Or (5t, [t), where is the noisy state and I, is a depth image. The depth sensor captures calibrated depth images with [640 480] resolution and 87 horizontal field-of-view, which mirror our sim-to-deploy experimental settings. Due to partial observability, the policy considers past information to decide the next action observation histories, H is the maximum history length. To enhance generalization, we integrate physical domain randomization and perception domain randomization as following:
((a (e,, O,; 0), where a, and o, are action and
Physies Domain Randomization. An environment appears geometry and is characterized by physics ex {ee=, ePa; ePm ,es, ePa) E IR_0, which is separated into environment properties and robot-environt inionmenviooment interactions. <P=: Dynamic friction, static friction, and restitution coefficients, affecting slipperiness; eP*; Gravity; eP': Mass, simulating extra burdens or flat tires; ePr; External forces; ePx: Discrepancies in actuator setpoints.
Perception Domain Randomization. Since velocity relies the inertial measurement unit (IMU), which is typically noisy [14], add Gaussian noise to simulate this: i ~ NNvv, o?) for linear velocity and w ~ N(w, oğ) for angular velocity, where and w are ground-truth in the simulator. Based on the empirical study [15], depth cameras suffer from precision and lateral noise, as well as invalid values at full sensor resolution (nan ratio). We model precision noise as A(O, t pi and lateral noise as A /V to + 4 0T(T// 0)), where is the measured depth of pixel (u, v) and its azimuth. The nan ratio modeled by a uniform distribution ((, 8man)
C Sim-to-Deploy Experiments
During training. the IsaacGym simulator runs at 200 Hz, and the policy runs at 50 Hz. In simulation experiments, ROS Gazebo provides odometry at 1000 Hz. In real-world experiments with Jackal, Faster-LIO [13] with IMU preintegration [14] fuses Lidar and IMU for 200 Hz odometry. Both settings use a depth camera RealSense D435i with [640 x 480) resolution at 30 fps, synchronized to odometry using ROS approximate time.
C.1 Simulation Training Setups
Uneven Terrain Datasets, To generate datasets, we utilize a digital elevation model (DEM) rep- resented as a raster obtained from real-world high-resolution topography dataset 2. The entire map is seamlessly divided into tiles of size [128 128) with resolution 0.1 m. 3000 training data for DDPM and 100 evaluation data for algorithmic performance randomly selected within specific geographical range, defined by latitude and longitude intervals: [33.5874, 111...166.0000 166.0058). [33.5874, - 1115.99911 [33.5929, - 11111.0556 [33.5929, -115.9991] For sim-to-sim deploy- ment experiment in ROS Gazebo, we get 30 terrains, each of size [320 320] with resolu-
16

---

tion 0.1 m, from latitude and longitude intervals: [44.1859, - 113.8802), [44.1881, 111.18883) [44.1881, -1 -113.8733, (44.1859, - -13.87733
Policy Parameterization. Our privileged (teacher) and deployment (student) policies use an encoder-decoder architecture. The encoder utilizes the first 16 layers of ResNet-18 [9] to extract feature representations from the terrain elevation or depth image. The deployment pol- icy employs neural circuit policy (NCP) [10], specifically the closed-form continuous time (CFC) network, to generate linear and angular commands. While Vanilla RNN, LSTM, GRU, and NCP achieve similar accuracy with suffi- cient learning steps, we choose NCP for its su- perior performance with significantly fewer pa- rameters [10]. We show details of PPO Ta- ble networks Fig. 6, and domain random- ization in Table 4. This setting keeps consistent for sim-to-sim and sim-to-real experiments.
coe 25e mull
State
[2 33% 2011
SEL betth
Fecond
Encoder
Noley
ro
Figure 6: The network architecture for the teacher (privileged) and student (deployment) policies, with N parallel robots training. The encoder uses the first 16 layers of ResNet-18, referred to ResNet-16.
Hyperparameter
Value
Hyperparameter
Value
Hardware Configuration
One RTX 4090
PPO Clipping Parameter
0.2 Adam
Action Coefficient
[2.0, 1.4]
Optimizer
Discount Factor
0.99
Learning Rate
5 10-4
Learning Epoch
2
Max Iterations
x 10
Generalized Advantage Estimation
0.95
Batch Size
5 x 10
Entropy Regularization Coefficient
0.005
Minibatch Size
10a
Table 3: Hyperparameters for Privileged Policy PPO.
Parameter
Type
Distribution
Curriculum Range
Environment
Dynamic Friction
Set
Gaussian
[0.1, 1.0
Restitution
Set
Gaussian
[0, 0.2
Gravity
Add
Gaussian
[0, 0.5
Robot
Mass
Add
Uniform
[0, 1.5]
Position
Add
Gaussian
[0, 0.05
Orientation
Add
Gaussian
[0, 0.01]
Angular Velocity
Add
Gaussian
[0, 0.1
Linear Velocity
Add
Gaussian
0.1
Robot-Euvironment
Interaction
Depth Precision
Add
Gaussian
[0.0015, 0.015]
Depth Noise
Add
Gaussian
[0, 0.01)
Depth Nan Ratio
Set
Uniform
0.3
External Foree
Add
Gaussian
[0. 0.5]
Actuator
Add Uniform
[0, 0.05]
Procedural Environment
Generation
Random Uniform
Set
Uniform
Height [- 0.45, 0.45], Step [0.005, 0.045]
Slope
Set
Uniform
[0.05, v3/2]
Discrete Obstacles
Set
Uniform
Height [0.4, 10). Size [0.1, 2.0), Num. [1
Wave
Set
Uniform
Num. [1, 20]. Amp. [0.1, 1 4/Num)
Table 4: Domain Randomization Parameters.
Additionally, we justify the procedural environment generation implemented in [11], where the height data type should be changed from integer to double accommodate our parameter ranges. In the Random Uniform terrain, "step" represents the maximum height difference between two adjacent elevation grids. We choose 0.045 m the upper step limit because it is consistent with Jackal's dimensions, where the chassis-togground distance is 0.058 based on our measurements. Wave
17

---

environment generates terrains based on the trigonometrie functions. Similarly, other procedurally generated environments are also navigable.
C.2 Sim-to-Sim Experiment
In addition benchmarking on the ClearPath Jackal robot, we train the ClearPath Husky robot using the same settings as the Jackal and tested in the same ROS Gazebo environments E-30 detailed in Sec. C.I. We evaluated across 30 environments, each containing 1000 pre-sampled start-goal pairs, resulting in total 30 000 trials. The results in Table 5 demonstrate that our method generalizes effec- tively across different wheeled platforms, maintaining an advantage over other methods in terms of success rate, orientation vibration, and position jerk. All methods performed better overall with the Husky, due to its better navigability on uneven terrains than Jackal. However, the performance dif- ferences between our method, Natural Adaptive Terrain (N-AT), and Procedural Generation Curricu- lum (PGC) relative other methods mirror those observed in the Jackal experiment, with challenges such perception noise and erratic prediction persisting for competing methods. In summary, method generalizes well across different wheeled platforms and maintains a performance advantage over baseline methods.
Succ. Rate (%)
Traj. Ratio
Orien.
Vib. (aa))
Orien. Jerk (aad)
Pos. Jerk
Falco [16]
53
1.47
1.85
91.6
33.6
MPPI [17]
65
1.32
0.74
145.5
33.7
TERP [18]
46
1.54
0.7
141.1
33.1
POVN [19]
43
2.5
1.28
121.4
31.6
N-AT
69
1.31
0.98
170.3
12.1
PGC
68
2.1
1.05
233.7
12
Ours
85
1.34
0.65
125.7
32
Table 5: Statistical results for Husky Gazebo simulation comparing our method with baselines of PGC and
N-AT as well as previous works. 30 environments have 1000 start-goal pairs in each. Green and Bold indicate the best and second-best results.
C.3 Sim-to-Real Experiment
Wheeled Robot Navigation. In the Jackal real-world experiment, we ablate the physics (w/o Physics) and perception (w/o Percept) domain randomization (DR) to study their contributions, shown in Table and Table 7. We observed varying contributions of physics and perception do- main randomization. First, in forest, mud, and gravel, removing either domain decreased the success rate, but these ablations still performed better than baselines due to ADTG. In arid and rust, the perception domain crucial because of increased perception noise. Second, we iden- tified drawbacks in the perception domain, particularly in dunes where the camera depth quality heavily depended on exposure, which is difficult to model. Last, without the physics domain, the robot showed improved orientation smoothness but struggled to make reactive behaviors, since the learned are movements in slippery areas benefited from this domain. Additionally, for computa- tional efficiency per frame, our method averaged 28.68 ms over 104 frames, compared to MPP['s 20.78 ms, Falco's 50.15 ms, and POVNav's 157, .2 ms.
Sue. Rate (%)
Traj.
Ratio
Orien. Vib, (add)
Orien.
Jerk rad
Pos.
Jerk (3)
R
D
R
D
R
D
R
D
Ours
80
45
1.25
1.32
0.11
0.13
113.3
73.4
20
18.1
Ours w/o Physics DR
74
42
1.18
1.35
0.1
0.13
112.6
74.2
25
20.3
Ours w/o Percept DR
48
1.27
1.31
0.11
0.12
117
72.5
24.6
17.9
Table 6: Statistical results for (orld- and D(une) comparing our method with ablations of physics and perception domain randomization. R: grass, forest, arid, rust, mud, and gravel; D: dune hard, tough, and expert. Each environment for each method has 36 start-goal pairs. Green and Bold indicate the best and second-best.
The issue of imprecise depth measurements highlighted in Fig. 7, where portions of the elevation map appear above the trajectories due to depth measurements inaccurately portraying unstructured objects. Compared to baselines that rely on elevation maps, our method showed more success. This is because the perception randomization during training introduces depth noises, allowing the
18

---

Success Rate (%)
Grass
Forest
Arid
Rust
Mud
Gravel
Dune-Hard
Dune- Tough
Dune- ExpeeeExertt
Falco [16]
100
25
58
33
39
28
31
17
17
MPPI [17]
100
22
92
56
86
69
75
17
11
TERP [18]
100
19
86
53
83
64
53
11
11
POVN [19]
100
14
42
Ours
100
33
100
58
97
92
75
39
22
Ours w/o Physics DR
100
28
97
58
78
81
72
36
17
Ours w/o Percept DR
100
25
89
50
81
89
72
39
33
Table 7: Detailed results of success among Falco, MPPL, TERP, POVNav, and method with ablations physies and perception domain randomization. Each environment for each method has 36 start-goal pairs.
Figure 7: Qualitative presentation of four real world experiments, with each featuring rectified RGB and depth images, accompanied by an elevation map marked with four distinct trajectories: [Green] Ours, [Orange] Falco, [Red] MPPI, [Blue] TERP. From left to right: [Arid] [Mud] Characterized by dense shrubhery, this scenario highlights method's is ability navigate slippery conditions and obstacles effectively. Tough] [Dune- Expert] Our method's preference for flatter regions proves
safer strategy an environment fraught with
unpredictable ravines.
policy be more robust against misleading depth measurements. Similar results are also observed recent work for legged locomotion [5]. The complex vehicle-terrain interactions posed another challenge. As shown in Fig. 7, method's trajectories had more curves, representing reactive behavior in slippery areas and leading the relatively large trajectory ratio in Table 1. The learned are movements are simple but effective, which will be explored further in our future research.
Quadruped Robot Locomotion. Refer to Parkour [12] Appendix B for detailed training in simu- lation. The parameters that differ from Parkour are follows, with all other settings remaining the same. We set the maximum iteration of PPO to 10 and the number of parallel environments to 200. For the environment, the terrain size is [128 x 128] with horizontal scale of 0.1. The nine ex- perimental environments with various challenges, including slippery surfaces, sloped terrains, and unpredictable ravines. These environments are the same those used in the ground vehicle ex- periment but were traversed continuously in a loop. The gaits of our ADTG and the built-in Gol MPC were natural, unlike the Procedural Generation Curriculum (PGC), which showed robust but prostrate postures. The PGC policy performed well but faced limitations due sudden movements (jumps) that caused the robot to roll over, as shown in Fig. PGC's issue may originate from its over-challenging unrealistic environments and propose investigate in the future.
Figure 8: Failure case of Procedural Generation Curriculum policy due to rapid movements.
19

---

Appendix References
[AI] Kappen, Hilbert J., Vicenç Gómez, and Manfred Opper. "Optimal control graphical model inference problem." Machine learning 87 (2012): 159-182.
[A2] Levine, Sergey. "Reinforcement learning and control as probabilistic inference: Tutorial and review." arXiv preprint 1rXivv1805.000 (2018).
[A3] Uchara, Masatoshi, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. "Understanding Reinforcement Learning-Basee Based Fine-Tuning of Diffusion Models: Tutorial and Review." arXiv preprint atXiv:2407.13734 (2024).
[A4] Ben-Hamu, Heli, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lip- man. "D-Flow: Differentiating through Flows for Controlled Generation." arXiv preprint arXiv:2402.1 14017 (2024).
[A5] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Ad- vances in neural information processing systems 33 (2020); 6840-6851.
[A6] Nichol, Alexander Quinn, and Prafulla Dhariwal. "Improved denoising diffusion probabilistic models." " In International conference on machine learning, pp. 8162-8171. PMLR. 2021.
[A7] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint Xii:11707.06347
[A8] Ross, S., Gordon, G., and Bagnell, D. (2011, June). A reduction of imitation learning and structured prediction to no-regret online learning. Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics (pp. 627-635). JMLR Workshop and Conference Proceedings.
[A9] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition (pp. 770- 778).
[A10] Lechner, M., Hasani, R., Amini, A., Henzinger, T. A., Rus, D., and Grosu, R. (2020). Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence, 2(10), 642-652.
[A11] Makoviychuk, Viktor, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller et al. "Isaac gym: High performance gpu-based physics simulation for robot learning." arXiv preprint arXiv:2108. 10470 (2021).
[A12] Zhuang, Ziwen, Zipeng Fu, Jianren Wang. Christopher Atkeson, Soeren Schwertfeger, Chelsea Finn, and Hang Zhao. "Robot parkour learning." arXiv preprint arXiv:2309.05665 (2023).
[A13] Bai, C., Xiao, T., Chen, Y., Wang, H., Zhang, F., and Gao, X. (2022). Faster-LIO: Lightweight tightly coupled LiDAR-inertial odometry using parallel sparse incremental voxels. IEEE Robotics and Automation Letters, 7(2), 4861-4868.
[A14] Y. Yu et al., "Fast Extrinsic Calibration for Multiple Inertial Measurement Units in Visual- Inertial System." 2023 IEEE International Conference on Robotics and Automation (ICRA), London, United Kingdom, 2023, pp. 01-07.
[A15] G. Halmetschlager-Funne,, M. Suchi, M. Kampel and M. Vineze, "An Empirical Evaluation of Ten Depth Cameras: Bias, Precision, Lateral Noise, Different Lighting Conditions and Ma- terials, and Multiple Sensor Setups in Indoor Environments." in IEEE Robotics & Automation Magazine, vol. 26, no. 1, pp. 67-77, March 2019.
20

---

[A16] Zhang, Ji, Chen Hu, Rushat Gupta Chadha, and Sanjiv Singh. "Falco: Fast likelihood-based collision avoidance with extension to numan-guided navigation." Journal of Field Robotics 37, no. 8 (2020): 1300-1313.
[A17] Williams, Grady, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou. "Aggressive driving with model predictive path integral control." In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 1433-1440, IEEE, 2016.
[A18] Weerakoon, Kasun, Adarsh Jagan Sathyamoorthy, Utsav Patel, and Dinesh Manocha. "Terp: Reliable plamning in uneven outdoor environments using deep reinforcement learning." In 2022 International Conference on Robotics and Automation (ICRA), pp. 9447-9453, IEEE, 2022.
[A19] Pushp, Durgakant, Zheng Chen, Chaomin Luo, Jason M. Gregory, and Lantao Liu. "POV- Nav: A Pareto-Optimal Mapless Visual Navigator" arXiv preprint arXiv:2310 11.2210.14000 14064 (2023).
21


Reviews:
[{"review_id": "aRzVutdHQ2", "reviewer": "Reviewer_rh2f", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces the Adaptive Diffusion Terrain Generator (ADTG), a novel environment generation method for training autonomous robots to navigate uneven terrains. This method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to create diverse and challenging terrains tailored to the robot's current capabilities. Unlike conventional heuristic-based methods, ADTG dynamically adjusts terrain complexity and variety based on the robot's evolving performance.", "summary_of_recommendation": "While the Adaptive Diffusion Terrain Generator (ADTG) presents an innovative approach to terrain generation, there are areas where the justification of the algorithm design could be strengthened.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Eq.1, the definition is not clear. The expectation is over what?(Policy or terrain parameters?)\n2. A deeper explanation of the design choices in ADTG, such as the selection of DDPMs over other generative models, and the specific weighting function used for blending terrains, would provide a better understanding of the underlying principles and their impact on performance.\n3. While the experiments demonstrate the effectiveness of ADTG, additional real-world validation across a broader range of scenarios and environments could further substantiate the claims.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CdmdPxgegk", "reviewer": "Reviewer_1ULr", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces Adaptive Diffusion Terrain Generator (ADTG), a two-stage approach to generating terrain maps for training policies in simulation (IsaacGym) for effective learning and robust deployment to other simulation (Gazebo), and real world. The first stage is to train a Denoising Diffusion Probabilistic Model from 3000 representative terrain maps based on real terrain data, and using this model to blend terrains from the initial dataset with generated maps with a weighting determined from policy success. The second is to add more diverse and complex terrains by manipulating the noise in the diffusion process.", "summary_of_recommendation": "Clearly a standout work with novel contributions and significant experimental results on hardware.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "See above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZJGEiw4Is4", "reviewer": "Reviewer_RP3g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a diffusion-based terrain generation and assignment algorithm. The author's theorise that by providing a terrain generator that is both realistic and flexible will improve policy performance when trained with reinforcement learning.  They ablate their methods with existing work that uses parameterised terrains (e.g. steps, perlin noise), and show that non-parametric diffusion generators outperform the parameterised methods. They further ablate their methods with existing navigation methods, and do extensive real world training. They propose a blending strategy to blend the latent space of two (or more?) terrain embeddings to get new terrains with varied traverse difficulties. They also propose adjusting the diffusion denoising steps to gradually increase diversity over time.", "summary_of_recommendation": "If the authors provide a justification for the baseline (PCG) parameters used or an ablation to show these are not unfairly disadvantaging the baseline, I would strongly recommend this paper for CoRL. Without this justification, I am uncertain but leaning towards 'accept' as many non-curriculum based baselines are compared and the contributions are valuable nonetheless.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Are the PCG parameters and terrain types appropriate for the Jackal? Are you able to verify this or ablate them? If the ADTG policy can't traverse a reasonable range of these PCG terrain parameters then these should be adjusted for fair comparison.  \n\n2. Are there limitations of your proposed method?\n\n3. Were the quadruped experiments ran in the same was as the Jackal experiments (sampling 36 start-goal pairs per experiment)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JLBL0lIrsA", "reviewer": "Reviewer_NMmZ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a model-free reinforcement learning approach proposing robust robot control policies for navigating complex terrains, relying on parallel physics simulations and environment generators. The Adaptive Diffusion Terrain Generator (ADTG) presented here addresses the limitations of traditional approaches of parameter-based heuristics by using Denoising Diffusion Probabilistic Models (DDPMs) to create diverse, complex terrains tailored to the policy's abilities. ADTG blends terrains with performance-informed weights and manipulates initial noise to generate suitable challenges and novel environments. Experimental results claim that ADTG-trained policies outperform those trained with traditional methods and natural environments.", "summary_of_recommendation": "I have concerns regarding the feasibility and scientific rigor of their methods, particularly in capturing the terra-dynamics of complex, deformable, and flowable surfaces like sand and mud. These interactions are nonlinear and influenced by hysteresis effects, making accurate predictions challenging. While the authors claim their method outperforms existing ones, it is unclear if this is due to the proposed control policy or other factors related to robot-terrain interactions. This area requires extensive investigations of granular physics, which may be beyond the scope of this work.", "technical_quality": 1, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Fig 1, fig 4 and fig 6 are small hard to grasp.\n\nIt is unclear how section 4 generates flowable and deformable surfaces such as sand and mud.\n\nSection 5.2:\nThe metrics are not clearly explained.\nThe details regarding the robot's parkours in both sim and exp subsections are not given. Also, I realized that the mud terrain does not seem to be wet in the video. \nLimitations are not addressed. \nThe conclusion section does not seem to reflect the key messages of the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper has been reasonably well-received by the reviewers.  The paper is relatively clear and well-motivated.  

The reviewers appreciated the diversity of different platforms this was tested on (with the exception of reviewer 2 pointing out a single wheeled platform - more wheeled platforms would be appreciated).

The comparison to other navigation algorithms, and the use of real robots for testing are other strong points.

Experimentation is generally good, with a number of different studies undertaken.

I have some reservations about how scalable this is - in that the terrain maps used are presumably of a given feature size, and thus would only produce maps of a similar feature size.  Can the authors please comment?

The testing section is weak.
-There is a lack of specifics of the testing and evaluation - e.g., what is a failure? Some of the experimental and testing settings are missing (see reviewer 2)

-There is a lack of forthright discussion on limitations

I encourage the authors to fully address the reviewer comments to increase chances of acceptance.  Specifically, addressing limitations, adding more detail to the experimental session, and showing more relevant information will help to increase scientific rigor.


Update:

The authors have responded well and improved their submission in response to reviewer comments.
Decision: Accept