DEAL: Diffusion Evolution Adversarial Learning for Sim-to-Real Transfer
Authors: Wentao Xu
Huiqiao Fu
Haoyu Dong
Zhehao Zhou
Chunlin Chen
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion evolution with adversarial learning
reinforcement learning
Sim-to-Real transfer
system identification
Primary area: reinforcement_learning
TL;DR: We propose a novel system identification framework for sim-2-real transfer in reinforcement learning that combines Diffusion Evolution with Adversarial Learning (DEAL) to iteratively infer physical parameters with limited real-world data.

Abstract: Training Reinforcement Learning (RL) controllers in simulation offers cost-efficiency and safety advantages. However, the resultant policies often suffer significant performance degradation during real-world deployment due to the reality gap. Previous works like System Identification (Sys-Id) have attempted to bridge this discrepancy by improving simulator fidelity, but encounter challenges including the collapse of high-dimensional parameter identification, low identification accuracy, and unstable convergence dynamics. To address these challenges, we propose a novel Sys-Id framework that combines Diffusion Evolution with Adversarial Learning (DEAL) to iteratively infer physical parameters with limited real-world data, which makes the state transitions between simulation and reality as similar as possible. Specifically, our method iteratively refines physical parameters through a dual mechanism: a discriminator network evaluates the similarity  of state transitions between parameterized simulations and target environment as fitness guidance, while diffusion evolution adaptively modulates noise prediction and denoising processes to optimize parameter distributions. We validate DEAL in both simulated and real-world environments. Compared to baseline methods, DEAL demonstrates state-of-the-art stability and identification accuracy in high-dimensional parameter identification tasks, and significantly enhances sim-to-real transfer performance while requiring minimal real-world data.

DEAL: Diffusion Evolution Adversarial Learning for Sim-to-Real Transfer
Wentao Xu, Huiqiao Fu", Haoyu Dong, Zhehao Zhou, Chunlin Chen'
Department of Control Science and Intelligent Engineering.
School of Management Engineering. Nanjing University, China.
[wentaoxu, haoyudong, zhzhou] @smail.nju.eduu bqfu @smail.nju..cuu..n cichen@nju..cu..cn
Abstract
Training Reinforcement Learning (RL) controllers in simulation offers cost- efficiency and safety advantages. However, the resultant policies often suffer significant performance degradation during real-world deployment due to the re- ality gap. Previous works like System Identification (Sys-Id) have attempted bridge this discrepancy by improving simulator fidelity, but encounter challenges including the collapse of high-dimensional parameter identification, low identifica- tion accuracy, and unstable convergence dynamics. To address these challenges, we propose a novel Sys-Id framework that combines Diffusion Evolution with Adversarial Learning (DEAL) to iteratively infer physical parameters with limited real-world data, which makes the state transitions between simulation and reality similar as possible. Specifically, our method iteratively refines physical parameters through a dual mechanism: discriminator network evaluates the similarity of state transitions between parameterized simulations and target environment as fitness guidance, while diffusion evolution adaptively modulates noise prediction and denoising processes to optimize parameter distributions. We validate DEAL in both simulated and real-world environments. Compared baseline methods, DEAL demonstrates state-of-the stability and identification accuracy in high- dimensional parameter identification tasks, and significantly enhances sim-to-real transfer performance while requiring minimal real-world data.
Introduction
In recent years, Reinforcement Learning (RL) controllers have achieved significant success in robotic control through iterative trial-and-error learning [[5]. This enables robust control of complex robots even in unstructured environments. However, such learning processes demand extensive data collection, and conducting training directly in the real world poses challenges related to efficiency and safety. comuulation-bace training offers practical alternative by enabling large-scale data generation in parallelized and risk-free environments (6) Despite this, policies trained in simulation often experience substantial performance degradation when deployed on real-world systems due to the sim-to-real gap.
To mitigate the performance degradation caused by the sim-to-real gap, previous research has explored various sim-to-real transfer techniques [7]. Among these, Domain Randomization (DR) is one most widely used approaches [8-12). It improves policy robustness by randomizing environmental and robotic physical parameters within predefined ranges during training and injecting noise into observations and actions. This paradigm aims to expose agents to sufficiently diverse simulated scenarios, thereby enhancing robustness and facilitating zero-shot transfer to reality. However, DR
"Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

often relies heavily on expert knowledge to define suitable randomization ranges, tends to produce overly conservative behaviors [13], and may lead to unstable or prolonged training. Recent advances in simulator fidelity optimization 24 have introduced few-shot approaches that leverage limited real-world data to learn target environment parameter distributions, thereby narrowing the reality gap and aligning simulator physical dynamics with the reality. Nevertheless, these approaches still encounter challenges including high-dimensional system identification collapse, low identification accuracy, inherent instability during the alignment process.
To address these challenges, we introduce a novel Sys-Id framework for sim-to-real transfer, which innovatively combines Diffusion Evolution with Adversarial Learning (DEAL) to narrow the reality gap. As diffusion models have demonstrated exceptional capabilities in image and video synthesis [23
27. existing works primarily utilize them as high- nigh-dimensional data generators. In DEAL, Diffusion
Evolution (DE) 28 is employed both as a high-dimensional optimizer for parameter search and as a powerful generator within a Generative Adversarial Network (GAN) 229 This dual role enables the generation of physical parameters that bring simulated state transitions closer those observed in the real world. Meanwhile, a discriminator network evaluates each evolving simulator by scoring its fitness based on the similarity between simulated state transitions and real-world demonstrations, thereby guiding the evolution of parameter distributions. DEAL builds upon this adversarial learning framework but avoids the need for costly real-world data to train a noise prediction module, as required in traditional diffusion models. Instead, it estimates optimal parameters by weighting the entire parameter population according to their fitness scores, predicts noise based on the current parameters and the estimated optimum, and approaches the target parameters through iterative denoising process. To further improve performance, DEAL introduces automatic noise adaptation to balance exploration and exploitation and applies parameter normalization reduce crossover errors caused by parameter scale imbalances. These enhancements enable DEAL to achieve stable and accurate high-dimensional parameter identification, significantly improving sim-to-real transfer with minimal real-world data and reduced computational cost.
In the experiments, we evaluate DEAL on five sim-to-sim tasks (AllegroHand, Humanoid, Go2, Cartpole, Ant) and two sim-to-real tasks (Cartpole, Go2). First, we evaluate DEAL's parameter identification capabilities, particularly in high-dimensional settings. Using policy trained with Uniform Domain Randomization (UDR), we collect demonstrations in target environment and conduct parameter searches with DEAL to redefine the simulator. The policy is then retrained in the enhanced simulator and its transfer performance is tested in the target domain. We further assess DEAL's adaptability by expanding the search scale and analyzing its dependence on target-domain demonstrations. Finally, we complete the challenging sim-to-real transfer task. Experimental results show that DEAL achieves ofate-of-the-ar stability and identification accuracy in high-dimensional parameter identification tasks, effectively bridging the sim-to-real gap with limited real-world data. In particular, the contributions of this work are threefold:
1. We introduce a novel methods, DEAL, which innovatively combines Diffusion Evolution with Adversarial Learning to narrow the reality gap.
2. We develop a automatic noise adaptation mechanism to balance exploration and exploitation, and propose parameter normalization framework to counteract search errors caused by parameter scale imbalances.
3. We demonstrate the effectiveness of DEAL in both sim-to-sim and sim-to-real tasks, showing superior performance compared to existing baselines.
2 Related Work
Transferring simulation-trainn policies to the reality in stable and cost-effective manner has long been a goal for sim-to-real research. Previous work has approached sim-to-real transfer from both policy adaptation and system identification perspectives.
Policy adaptation optimization focuses on training policies that are robust to dynamics discrepancies without relying on real- wallwwrldd fine-tuning. Domain randomization (DR) is a foundational technique that injects randomness into simulation parameters to encourage generalization 10 [3] However, DR often requires manual tuning of randomization ranges, leading to suboptimal performance or
excessive training cost. Recent advancements mitigate these limitations by integrating curriculum
learning [30] or GAN-guided subspace prioritization [31), Dropo further improves upon DR by

---

leveraging off- off-policy data or human demonstrations to learn more effective randomization distri- bution .T.. Other methods introduce latent adaptation modules [2 33 and employ Concurrent Policy Optimization (4 to estimate environment variables in real time, but these approaches struggle with high-dimensional or partially observable dynamics. Domain adaptation techniques, such as anage-to-image translation [35], grounded action transformation (GAT) [[]] and ASAP [39], aim to align simulation-reale reality action spaces or visual inputs. However, they typically require costly real-world data and often lack generalizability across tasks.
System identification optimization calibrates simulation parameters match real-world dynamics using limited has focused on identifying simulation parameters that better align with real-world dynamics. Bayesian approaches like BayesSim [2]] and its extensions [22, 43
trajectories. Inspired by classical Sys-ID frameworks [2), recent work
formalize system identification as an inference problem, iteratively estimating posterior distributions over simulation parameters. However, these methods suffer from high computational costs and poor scalability in high-dimensional parame- ter spaces. Some data-efficient approaches adopt alternative strategies, ASID [9] designs exploration policies to collect informative real-world trajectories, while TuneNet [5 employs supervised learn- ing to map simulation trajectories to parameter gradients, bypassing iterative optimization. RL-based methods reframe the parameter search problem as a policy learning task, where trajectories serve as states and parameters as actions [17 44). these methods struggle with sparse rewards in complex tasks. For dynamics alignment, adversarial training frameworks like SimGAN (18) and EASI [14] utilize discriminator to distinguish simulated and real-world state transitions. EASI optimizes by selecting fixed elite parameters through manual sorting in evolutionary search, ignoring the information of the entire population. This reduces parameter population diversity and leads to a tendency to get trapped in local optima, which limits both the amount and accuracy of the searched parameters and often produces multiple sub-optimal solutions in system identification. In contrast, DEAL employs soft probabilistic weighting over the entire parameter population, preserving diversity and improving convergence stability. Its iterative denoising process continuously refines parameters without relying on noise prediction module, enabling stable and precise high-dimensional parameter search with minimal real-world demonstrations and lower computation cost, while effectively enhancing transfer capabilities in complex robotic tasks.
Approach
In this section, first provide brief background DE and GAN, then describe in detail how DEAL integrates DE with GAN, and demonstrate how it narrow the reality gap by aligning simulator's physcial parameters using limited real-world data.
3.1 Background
operate in two phases: diffusion and denoising. In the diffusion phase, Gaussian noise is progressively added to the original data, while a noise prediction module eg is trained to predict the added noise. The denoising phase iteratively estimates the original data and performs directed denoising to recover it. Diffusion Evolution (28) (DE) reinterprets this framework through evolutionary principles: denoising mimics evolution, diffusion emulate reversed evolution and add random perturbations act as mutations. This algorithm uses the iterative denoising process from diffusion models to refine solutions in parameter space, instead of recovering data distribution, it shifts an initial random population toward an optimized solution distribution. Similar to the relationship between energy and probability in statistical physics, evolutionary tasks can be connected to generative tasks by mapping fitness to probability density, diffusion models are directly predicting the original data To from noisy versions of 2g each time step, DE can estimate the optimal point by weighting the current population individuals x based on the corresponding fitness probabilities:
Diffusion Evolution Diffusion models 24 25
(o a, = Σ gif(z))N(ax Vaga, 1 -a,),, aEX,
where Ft is the corrupted data at timestep ~ [0, T). diffusion schedule ar governs the noise x is sample from the current data distribution X,, the probability mapping function g(-), typically implemented as Softmax, transforms fitness evaluated by f(() into probabilities, and the Gaussian distribution indicates the conditional probability of the current data point given any sample point as
3

---

the optimum, the weighted results of these are normalized by Z. Given the design of the diffusion process, ie., 2₁ varzo + vI-a,, a,e, the noise e can be estimated without the need for noise prediction module ep by:
E(x,, a. Itt v you (,
v1 04
Under DDIM 25 framework, this step-by-step denoising process can be described as:
It-1 V 10 v V1-at-1 a - oft owl,
where w is the random perturbations controlled by the noise schedule oe in denoising phase. More proofs about DE can be found in the Appendix A.I
Generative Adversarial Network The Generative Adversarial Network (GAN) 29 framework
employs two neural networks -a generator G and a discriminator - that engage in an adversarial training process to synthesize data matching the statistical properties of the training distribution. These networks optimize through a minimax game defined by:
min max G) = (x) G
+ Ezop, (z) - D(G(2)))
(4)
D
where (=(2) denotes the noise prior distribution, and represents samples from the real data distribu- tion Pdata- Through iterative updates, G learns to generate synthetic data indistinguishable from real samples under D's evaluation, thus achieving distributional alignment. For specialized objective, we use WGAN 455 in this work, the discriminator optimization objective is formulated as:
max E (s.aa.')~=d [(s,, a, s')] -(s..."''''"" D
(D(s, (D(s, a, s,))
(5)
where dS and d' denote trajectories from the simulator and reality respectively, this objective incentivizes D to discriminate between simulator and reality state-action transitions dynamics. The discriminator used in DEAL adopts fully connected MI.P with input (s, a, a,s'), ), two hidden layers of 256 units with ReLU activation, and scalar output. To further stabilize the training process, use Weight Clipping [45] to located the weights of discriminator a compact interval to satisfy the Lipschitz continuity condition, ensuring the effective computation of the Wasserstein distance.
3.2 DEAL
The schematic overview of the DEAL architecture is shown in Fig. and the pseudo-code is shown in Algorithm [ Through adversarial co-optimization, this framework progressively minimizes the distribution discrepancies until convergence, achieving a calibrated simulator that becomes indistinguishable from the reality the discriminator.
Diffusion Evolution Adversarial Learning In this work, by modeling the processes in reality and simulation as MDPs, the system identification objective narrow the gap between the state transitions of the target (reality) and source (simulator) domains. Due to different physical dynamic parameters, there's a reducible reality gap between the target domain state transition distribution P.(targget) and the source domain one (0). 0 denotes the physical parameters of the robot or the external environment, as detailed in Appendix A.7 Our method aims to search parameter distribution which is modeled Gaussian distribution to minimize this gap. Specifically, the goal of DEAL is to minimize the discrepancy between state transition distributions from different domains as follows:
min DEl4
| (Otargett (x)))
(6)
The similarity of state transitions between parameterized simulations and reality can be measured by fitness function f(-). which is replaced by a discriminator network in this work. The discriminator evaluates how similar the state transitions
the trajectories are to those the target domain and gives a corresponding score based on the level of similarity. With DE serves as the generator and combined with Equation (5), the overall search objective can be described as:
arg min max E(s...s')<d' (Btorget .n) [(ss, a, s')] (f.. D a,
(7)
BEU D

---

where m8) represents the trajectory collected by To in the simulator parameterized by 8. In this adversarial process, DE acts as the generator to generate and optimize the making the simulation trajectory increasingly similar to the real world trajectory d' (flarget, mo) from the discriminator's perspective, while the discriminator aims to distinguish between them as much as possible.
In this frame, firsty, we randomly sample physical parameters or from the initial search range U to initialize each simulator, then use the policy wo trained via UDR to collect trajectories that reflect the current environment's dynamics, After sampling state-action transition (s, a, s') sequence 6² and b from the target and source domain trajectories respectively, then fed them to the discriminator to evaluate each parameterized simulator's 's fitness, which represent the similarity of state transitions between the evolving simulator and target environment, it used to update discriminator according to Equation (5) and guide DE for parameter updates. During the parameter update phase. by mapping the fitness through Softmax to obtain a probability distribution, DE can estimates the optimal parameters do by weighting the current population individuals based on the corresponding fitness probabilities and predicts noise used for directed evolution. Finally, according to Equation (3) the next generation of parameters can be generated by:
02-1 vo + V a4-1 - off oww.
(8)
Discriminator
Pitnoss
Evaluate
DE
Target
Source
Extimade
A.A.
Rollout
DoneEse
Notse
.AA
Policy
Eatimnde
h Votae
Prnucise
M
0g
Optimiaation
Porameters
Figure 1: Schematic overview of DEAL. The framework iteratively optimizes parameters through dual mechanism: discriminator evaluates the similarity of state transitions sampled in source domain trajectories Ts and target domain trajectories as fitness, while DE estimates the optimal parameter On based on the fitness probabilities, adaptively updates noise predictions é and performs denoising to optimize parameter distributions until convergence.
Automatic Noise Adaptaion and Parameter Normalization To encourage DEAL to search for the optimum. use Equation (9) to finetune the noise impact factor ė base on fitness, then we let o; da, to adaptively adjust the noise schedule d,, which can influence the current random perturbation in search process. Specifically, this method adjusts the level of w to balance exploration and exploitation, when the parameter population has low fitness, it increases w to encourage exploration and generate more mutations in search of better optima. Conversely, when fitness is high, indicating the population is in good evolutionary state, it reduces w to focus on exploitation of the current optimal region.
= A exp ex (-E(g.a.s)>dd (s.a.s) [D a, s')) and à; do.,
where d represents trajectories collected in source domain and À is the influence coefficient, D the discriminator quantifies the fitness of each evolving simulator. Combined with Equation 8 the
5

---

Algorithm 1: DEAL
Input: Population size N, Parameter dimension K, Search steps T, Sample batch size M, UDR policy To. Search range 14, Target domain demonstration DT, Source domain demonstration Ds, Diffusion schedule or, Noise schedule ds Noise factor 6, Random perturbations w, Probability mapping function g(-).
Output: Optimal parameter distribution 0 1
1 Initialize parameters: 0² = 10 04 0 - u
2 for & [ 1,.,, 1] do
Vj E [1, Store Rollout(To- Sim (0) in Ds
3
for update and clip network weights step = 0. 1,2, ... , do
67= (s,, a,, 5,) + Cample(DT)) 6$ (s,, a, s s + Sample (Ds))
Update D according Equation 5 using bd and b
end
8
Evaluate fitness and Map to Probability:
Vj []: fitness = EE, T Rolloott(mm Sim(0))))) [D a, s')], Pj g(fitness)
Calculate noise factor 5 and Adjust noise schedule or according Equation 9 for env j = 1,2, .... N do
11
Z + Σ E== Pi -N(( vagl( ,1-11 ar)
8. + Σ Pi -N(P vai , a1)
13
8 0. Varbu
t- NN0,1 1k)
14
v1 at
0 + 1a + V1 a--1 ofē + 6,W
15
16
end
end
generation equation can be rewritten as:
0.-1 = vo +
a1-1 - -,f2 + G,w.
(10)
This mechanism balances exploration and exploitation for DEAL, encouraging thorough parameter space search while converging toward high-fidelity simulations.
have crossover errors due to the parameter scale imbalances. For instance, when searching for motor stiffness and damping, their values can differ by tens to hundreds of times, such parameters have unbalanced values and the optimization step lengths don't 't match may lead crossover errors during system identification. In this work, we nor- malize the parameters within their initial search range 41 this approach allows for percentage based adjustment of parameters, effectively reducing search errors caused by parameter scale mismatches.
Previous learning based Sys-Id methods 14
16 18
Experiments
In this section, compare DEAL with several baselines across multiple tasks to demonstrate its effectiveness. The key aspects are as follows:
1. We search for parameters in multiple tasks, including high-dimensional scenarios, then compare the search errors of DEAL and other baselines.
2. After enhancing the simulator across various tasks, we evaluate the transfer performance of the retrained policies using DEAL and other baselines.
3. We search for parameters with larger search scales to examine the search adaptability and assess the demonstration requirements of DEAL and other baselines.
4. Finally, we complete the challenging sim-to-real transfer and demonstrate the performance improvements brought by DEAL.

---

Figure 2: Experiment tasks in simulation (1~5): AllegroHand, Humanoid, Go2, Cartpole and Ant. Experiment tasks in reality (6~7): Cartpole, Go2. Presented in order from left to right.
Task Setup We test DEAL in 5 sim-to-sim tasks (A (AllegroHand, Humanoid, Go2, Cartpole, Ant) and 2 sim-to-real task (Cartpole, Go2), all environments are shown in Fig. 2 Hlegorgdd environment is a AllegroHand robot designed to interact with cube, it features 16 DoF distributed across the hand and fingers. The task is to use the hand to manipulate an object to align its orientation with the target orientation. (2) Humanoid environment contain a humanoid robot has 21 DoF, with 3 the abdomen, each for the right and left legs, and each for the right and left arms. Its tasks include walking stably and performing various movements to achieve specific speeds/directions, (3) Go2 environment is a 12-DoF Unitree GO2 quadruped, with each leg having 3-DoF. The task is to follow randomly chosen X. y, yaw target velocities and climb platforms. (4) Cartpole environment includes an inverse pendulum connected to 1-DoF cart. The task is to keep the pendulum on the cart balanced for as long as possible. (5) Ant environment is 8-DOF quadrupedal robot consisting of four legs attached to a common base. The task controlling the ant run as fast as possible.
Experiment Detail Our experiments employed Isaac Gym 46 as the simulator. In Isaac Gym, we can parallelly collect trajectories hundreds of environments with different parameters which means could evaluate hundreds of parameters parallelly. During parameter search, instantiat 200 aligument across varying physical parameters.
parallel environments to assess reality-to-simutee For Cartpole, Ant, Humanoid and AllegroHand, implement Soft Actor-Critic (SAC) [47 train a neural network as RL controller. For Go2, we adopt RMA 33 to develop controllers tracking velocity commands and climbing platforms, and train canter controllers using Ess-InfoGAIL [48] for bio-inspired running and command following. In our experiment, running on a PC equipped with Intel 55-04400KF and RTX 4060 Ti, DEAL can complete the entire search process within few minutes, the search computation cost can be found in the Appendix A.2 Additionally, we provide the simulation data budgets for each task, the detailed numbers shown Appendix
Baselines We design various baselines to demonstrate the efficiency our method. (1) EASI [4):
A learning-based Sys-Id Id method combines CMA-ES (49 and GAN [29] (2) Bayes Optimization (BO): Modification of Bayesian Sys-Id approach- BaaaachhhBBRnnnn (2 in search tasks with its return eval- uated by discriminator. (3) DEAL FN: DEAL without using Automatic Noise Adaptation base on fitness (4) PP: DEAL without using Parameter Normalization (5) Uniform Domain RandonizationnUDRR Uniformly sampling parameters from if at the beginning of each training it- eration. (6) Oracle: Direct training in the target environment in sim-to-sim experiments, representing the ideal upper bound for sim-to-sim transfer tasks.
FFN:
4.1 Parameter Search Capability
In this experiment, we selected five tasks and conducted thorough parameter search for each. The parameter list is detailed in the Appendix A.7 and mainly covers friction coefficients, restitution coefficients, masses of rigid bodies and motor properties such as stiffness, damping and friction. unknown, typically train a policy across wide range
When the parameters of target domain of simulation parameters in hope that the policy can thus handle possible rorldworld variations in dynamics or observations. In this experiment, the UDR policy To of Ant and Go2 trained within the range of x1/3xf,3 0,,3 33 x0,,, the remain tasks were trained within the range of [1/5 1/,500,.5 x fy), where 0, denoting the parameters of the target domain. The initial search range lI for parameters of each task is the same as its training range, we performed 50 steps of parameter search for each task with limited 'real-world" demonstration collected by mo. To avoid randomness, we presented the experimental outcomes using the average values from multiple random seeds. The evaluation metric is the average percentage error of the searched parameters relative to the true parameters, defined as follows:
0 glarget
Ep E
(11)
() (frrge
7

---

641
Ant with 11 params
Cartpole with params AllegroHand with 125 params
Go2 with 24 params
Humanoid with 92 params
Figure 3: Average search errors for each method, consisting of the error percentage of all parameters.
As shown in Fig. when comparing the final search results of the three methods with the target parameter errors, it is evident that DEAL exhibits significantly higher search accuracy than other baselines across all environments, especially in -igh-dimensional environments. The baselines often suffer from high-dimensional parameter identification collapse and fail to converge to precise solutions. Despite using less computation cost and the same demonstration in this experiment, DEAL achieve optimal performance which demonstrates its powerful search capability and stable performance.
4.2 Sim-to-Sim Transfer
In this experiment, we utilize the simulator updated from the previous experiment for new policy training. then transfer these newly trained policies to target environment to compare the performance with UDR and Oracle. Notably, the target environment in this experiment is the simulator set with the target parameter. This done compare the performance improvements brought by various enhanced simulators to the policy. As shown in Fig, left, the simulator improved by DEAL delivers the greatest performance boost to newly trained policies for all tasks, it close or only slightly inferior to the performance of Oracle whose policy is trained directly in the target domain. Moreover, in high dimensional tasks like Humanoid and AllegroHand, due to the high- high-dimensional parameter identification collapse of the baselines, the simulator undergoes negative optimization, the policies trained such simulator perform worse than those trained by UDR and may even cause training to crash, fail to converge, and lead to abnormal behavior. For Go2 task, after training an initial policy
175
Cartpole
Ant
Go2 Base Velocity
150
Diak
Basi Ln (mi))
100
Do
Ateraage
25
0.29
810
(MLes
1.
(Xies)
2.3
1sl
130-
AllegroHand
2500
Humanold
Go2 Base Velocity
100
2000
Bose Ln vall (ma))
15:00
Acanage
12:00
Stepo (4166)
Steps (00
Oracle
DEAL
EASI
B0
UDR
Command X- [0.52-3] Command (0.5.1.4
The average return in the target environment for the policies trained at each stage with DEAL and other sim-to-real baselines. Right: The speed tracking display of Go2 under training with DEAL and other sim-to-real baselines. up: speed in the robot's x-axis direction, down: speed in the robot's y-axis direction.
Figure 4: Left:
to collect torget-domain trajectories, we retrain the policy the simulator optimized by DEAL and other baselines. Then, we compare the adaptability of the final policy in target environment, this experiment examines the speed-tracking capability of policies, whether the controller can control the Go2 robot move forward in the specified direction and speed, and to ascend steps. As shown

---

in Fig. riight DEAL achieves the best performance the robot's movement accurately tracks the speed commands in both forward and vertical directions. However, the policies derived from other baselines fail to sustain tracking of velocity commands when they are progressively increased, ultimately leading to more severe speed collapse and destabilizing oscillations in robot motion than UDR. This experiment demonstrates DEAL significantly enhances transfer performance across all tasks.
4.3 Parameter Search Adaptability and Data Requirements
Table 1: Average search error percentage (x 100%) (See Appendix A.8
for error bars).
Cartpole
Humanoid
AllegroHand
Method
=
{ {=15
[ =
{=10 10
{ 15
{ 2=
=
{ =15
{ ==20
DEAL
0.85
1.74
2.63
0.81
1.65
2.50
0.83
1.69
2.57
DEALXPN
1.90
2.85
3.76
2.40
3.53
4.58
2.60
3.80
4.92
DEALAFN
1.67
2.96
4.28
1.57
2.79
4.03
1.59
2.80
4.06
EASI
4.03
6.12
8.40
2.94
4.64
6.03
4.25
6.82
9.32
BO
5.20
8.52
11.78
4.67
7.29
10.06
3.75
6.32
9.16
In this experiment, we search for parameters on larger search scales and determine the require- ments for demonstration of DEAL and other baselines. As shown in Table ! denoting 0, as the target parameters, DEAL attain the minimal search errors in the initial search ranges
of lgg 0e. [ x 8:] across all tasks, and maintain estimation accu- racy even under severely limited parameter distribution priors and
Parmeneter Search Eiror (g- 6)
extreme initial values during large-scale searches. Notably, UDR policies trained with a range of I Og. 10 0,] have started to fail
3:20
DES.
To in the target domain, producing low quality even all failed trajecto-
ries that cause other search baselines' errors to surge. Such scenarios are prevalent in real world deployment the initial UDR policies rarely perform well at the outset. This highlights DEAL's strong search adaptability and stable performance. Furthermore, due to the lack of parallel data collection in reality, the scarcity of data and the high cost of collection mean few shot methods must minimize data requirements. Fig. 5 reveals DEAL's superior data efficiency: While existing Sys-Id methods suffer significant performance degradation with sparse trajectory samples, DEAL sustains robust identification accuracy under data-scarce conditions.
100
Densonttiion Langt
Figure 5: Average search er- rors of DEAL and other base- lines when given different demonstration lengths in Hu- manoid task.
4.4 Sim-to-Real Transfer
In this experiment, we first deploy the UDR-traine policy to collect real-world trajectories in few seconds, then optimize the simulator with DEAL, retrain the policy and redeploy to compare sim-to- real performance. In the Cartpole task, the controller outputs target cart position to maintain the pole in balance. Stability measured by pole-angl errors and cart-velocity, both of which are reduced after DEAL optimization shown in Table 2 demonstrating improved sim-to-real performance. In the Go2 environment, the controller out- puts target position signals for each joint at a frequency of 50 Hz. These signals are then
Table 2: Cartpole sim-to-real performance.
converted into motor torques by the motor's built-in PD controller operating at 1000 Hz and applied to the corresponding joints. In this section, we train a climbing policy based
Method
Angle Error x x10
Cart Vel x10- 1.480 1.214
UDR DEAL
3.655:1.112
1.372:10.382
on RMA (33] for the experiment of climbing onto high platform, and train a canter policy using Ess-InfoGAN [48] for the quadruped robot's 'ss rumning experiments. Firstly, we conduct repetitive high-platform climbing experiments to evaluate the policy performance. As shown in Fig. 6 both
UDR and Unitree's built-in RL policies are unable to reproduce the climbing behavior trained in simulation, always fail control Go2 to climb the platform. After optimized by DEAL, the Go2 robot can fluently and successfully complete the task, the probability of success has been significantly

---

improved. Subsequently, we deploy the canter policy and compare the running performance of the controllers trained with UDR and DEAL. As shown in Fig. 7 the UDR policy fails to reproduce the straight-running gait observed in simulation, resulting in significant deviations in both the magnitude and direction of the velocity. contrast, the policy retrained with DEAL. accurately tracks the velocity commands and maintains forward speed close to that achieved in simulation during straight-line locomotion. These results indicate DEAL indeed narrows the sim-to-real gap and enhances the performance of the retrained policy in reality.
A
Figure 6: (A) Deploy with the climbing controller after being optimized with DEAL.(B) Deploy with the climbing controller trained with UDR. (C) Deploy with Unitree's built-in RL controller.
COMMAND
DEAL
UDR
Figure 7: (A, C) Deploy with the canter controller after being optimized with DEAL. A: side-view imaging, C: top-view imaging. (B, D) Deploy with the canter controller trained with UDR. B: side-view imaging, D: top-view imaging.
5 Limitations
Although DEAL demonstrates remarkable performance in high-dimensional parameter optimiza- tion tasks under limited demonstration data, its applicability may face challenges environments with diverse state transition distributions. This potential limitation arises from the inherent binary classification nature the GAN discriminator, which designed to distinguish between one target environment transition and simulator transition time. As result, the discriminator may struggle to provide multimodal fitness evaluations during optimization, suggesting an area for future improvement handling more complex multimodal environmental dynamics.
6 Conclusion
In this work, we propose DEAL, novel Sys-Id framework that combines diffusion evolution with adversarial learning for sim-to-real transfer. DEAL optimizes simulator parameters through dual mechanism: discriminator evaluates the similarity of state transitions between evolving simulators and reality as fitness guidance, while DE adaptively refines parameter distributions to narrow the reality gap by using fitness-driven denoising. Extensive experiments in simulation and real-wor world demonstrate DEAL's superior performance in high-dimensional parameter identification and robust transfer performance across challenging tasks, outperforming baselines with minimal real-world data and lower computation cost. We believe that DEAL advances sim-to-real transfer and offers promising approach for deploying RL controllers in the real world.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the Major Science and Technology Project of Jiangsu Province under Grant BG2024041, National Key Research and Development Program of China under Grant 2023YFD2001003, and the Fundamental Research Funds for the Central Universities under Grant 011814380048.
References
[1] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, "I learning agile and dynamic motor skills for legged robots," Science Robotics, vol. no. 26, p. caau5872, 2019.
[2] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, "Learning robust pereeptive locomotion quadrupedal robots in the wild," Science robotics, vol. no. 62, p. eabk2822, 2022.
[3] E. Kaufmann. L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza, "Champion-level drone racing using deep reinforcement leamning" Nature, vol. 620, no. 7976, pp. 982-987, 2023.
[4] D. Hoeller, N. Rudin, D. Sako, and M. Hutter, "Anymal parkour: Learning agile navigation for quadrupedal robots," Science Robotics, vol. 9, no. 88. p. eadi7566, 2024.
[5] T. Miki, Lee, L. Wellhausen, and M. Hutter, "Learning to walk in confined spaces using 3d representation." in 2024 IEEE International Conference on Roboties and Automation (ICRA). IEEE, 2024, pp. 8649 8656.
N. Rudin, D. Hoeller, P. Reist, and M. Hutter, "Learning walk in minutes using massively parallel deep reinforcement learning." in Conference on Robot Learning. PMLR, 2022, pp. 91-100.
W. Zhao, J. P. Queralta, and T. Westerlund, "Sim-tt-real transfer in deep reinforcement learning for robotics: survey," in 2020 IEEE symposium series on computational intelligence (SSCI). IEEE, 2020, pp. 737- 334.
[8] X. B. Peng, M. Andrychowicz, Zaremba, Abbeel, "Sim-to-real transfer robotic control with dynamics randomization," in 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 3803-3810.
[9] J. Tobii,, R. Fong, A. Ray, J. Schneider, W. Zaremha, and P. Abbeel, "Domain randomization for transferring deep neural networks from simulation to the real world," in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 23-30.
[10] S. James, A. J. Davison, and E. Johns, "Transferring end-to-end visuomotor control from simulation to real world for multi-stage task," in Proceedings of the Ist Annual Conference Robot Learning, ser. Proceedings Machine Learning Research, Levine, V. Vanhoucke, and K. Goldberg. Eds., vol. 78. PMLR, 13 15 Nov 2017, pp. 334-343 343.
[11] G. Tiboni, K. Arndt, and V. Kyrki, "Dropo: Sim-to-real transfer with offline domain randomization." Robotics and Autonomous Systems, vol. 166, p. 104432, 2023.
[12] Dao. Green, H. Duan, A. Fem, and Hurst, "Sim-to-real learning for bipedal locomotion under unsensed dynamic loads." in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 10449-10455.
[13] O. M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, "Learning dexterous in-hand manipulation." " The International Journal of Robotics Research, vol. 39, no. 1, pp. 30 2020. [Online]. Available: httpss//doo..orgg00117777/273322
[14] H. Dong, H. Fu, W. Xu, Z. Zhou, and C. Chen, "Easi: Evolutionary adversarial simulator identification for sim-to-real transfer," Advances Neural Information Processing Systems, vol. 37, pp. 6603 6624, 2024.
[15] A. Allevato, E. S. Short, M. Pryor, and A. Thomaz, "Tunenet: One-shot residual tuning for system identification and sim- to-ato-real robot task transfer." Proceedings of the Conference on Robot Learning. ser. Proceedings of Machine Learning Research, L. P. Kaelbling, D. Kragic, and K. Sugiura, Eds., vol. 100. PMLR, 30 Oct-01 Nov 2020, pp. 445-455.
[16] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox, "Closing the sim-to-real loop: Adapting simulation randomization with real world experience," in 2019 International Conference Robotics and Automation (ICRA), 2019, pp. 8973-887 8979.
11

---

[17] Y. Du, O. Watkins. T. Darrell, P. Abbeel, and D. Pathak. "Auto-tuned sim-to-real transfer," in 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 1290-1296.
[18] Y. Jiang. T. Zhang, D. Ho. Y. Bai, C. K. Liu, S. Levine, and J. Tan, "Simgan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning," in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 2884-2890
[19] M. Memmel, A. Wagenmaker, C. Zhu, P. Yin, D. Fox, and A. Gupta, "Asid: Active exploration for system identification in robotic manipulation." " arXiv preprint arXiv:2404. 12308, 2024.
[20] P. Huang. X. Zhang, Z. Cao, S. Liu, M. Xu, W. Ding. J. Francis, B. Chen, and D. Zhao, "What went wrong? closing the sim-to- to-rcal gap diffcrentiable causal discovery," in Proceedings of The 7th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, J. Tan, M. Toussaint, and K. Darvish, Eds., vol. 229. PMLR, 06-09 Nov 2023, pp. 734-760.
[21] F. Ramos, R. C. Possas, and D. Fox, "Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators," arXiv preprint arXiv: 1906,0172,, 2019.
[22] F. Muratore, C. Eilers, M. Gienger, and Peters, "Data-efficient domain randomization with bayesian optimization," IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 91 1-918, 2021.
[23] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and Ommer, "High-resolution image synthesis with latent diffusion models," in Proceedingy of the IEEE/CVF conference on computer vision and puttern recognition, 2022, pp. 10684-10695.
[24] J. Ho, A. Jain, and P. Abheel, "Denoising diffusion probabilistic models," Advances in neural information processing systems, vol. 33. pp. 1840-6851, 2020.
[25] J. Song, C. Meng, and S. Ermon, "Denoising diffusion implicit models," arXiv preprint arXiv:2010.02500 2020.
[26] G. Batzolis, J. Stanczuk, C.-B. Schönlieb, and C. Etmann. "Conditional image generation with score-based diffuusin models." arXiv preprint arXiv:2111.33600 2021.
[27] D. Epstein, A. Jabri, B. Poole, A. Efros, and A. Holynski, "Diffusion self-guidance for controllable image generation." Advances in Neural Information Processing Systems, vol. 36, pp. 16222216 16--16627 239, 2023.
[28] Y. Zhang, B. Hartl, H. Hazan, and M. Levin, "Diffusion models are evolutionary algorithms," arXiv preprint arXiv:2410.02543 2024.
[29] I. J. Goodfellow, J. Pouget-Abadics M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," Advances in neural information processing systems, vol. 27, 2014.
[30] I. Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q Yuan, W. Zaremba, and I.. Zhang, "Solving rubik's 's cuhe with rohot hand," 2019.
[31] B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull, "Active domain randomization," in Proceedings of the Conference on Robot Learning, ser. Procoedings Machine Learning Research, Kaelbling. D. Kragic, and K. Sugiura, Eds., vol. 100. PMLR, 30 Oct-01 Nov 2020. pp. 1162-1176.
[32] I. M. A. Nahrendra, B. Yu, and H. Myung, "Dreamwaq: Learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning." 2023.
[33] A. Kumar, Z. Fu, D. Pathak, and J. Malik, "Rma: Rapid motor adaptation for legged robots," 2021.
[34] G. Ji, J. Mun, H. Kim, and J. Hwangbo, "Concurrent training of a control policy and state estimator for dynamic and robust legged locomotion." IEEE Robotics and Automation Letters, vol. 7, 2, pp. 4630-4637, 2022.
[35] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L.. Downs, J. Ibarz, P. Pastor, K. Konolige, S. Levine. and Vanhoucke, "Using simulation and domain adaptation improve efficiency of deep robotic grasping." in 2018 IEEE International Conference on Robotics and Automation (ICRA). 2018, pp. 4243-4250
[36] J. P. Hanna and P. Stone, "Grounded action transformation for robot learning in simulation." in Proceedings of the Thirty-Firs First AAAI Conference on Artificial Intelligence, ser. 17. AAAI Press, 2017, 3840.
12

---

[37] H. Karnan, S. Desai, J. P. Hanna. G. Warnell, and P. Stone, "Reinforced grounded action transformation for sim-to-real transfer," 2020.
[38] S. Desai, I. Durugkar, H. Karnan, G. Warnell, J. P. Hanna, and P. Stone, "An imitation from observation approach to transfer learning with dynamies mismatch." " in Proceedings the 34th International Conference on Neural Information Processing Systems. Red Hook. NY, USA: Curran Associates Inc., 2020.
[39] T. He, J. Gao, W. Xiao, Y. Zhang, Z. Wang, J. Wang. Z. Luo, G. He, N. Sobanbab, C. Pan al., "Asap: Aligning simulation and real-world physies learning agile humanoid whole-body skills," arXiv preprint arXiv:2502.01144 2025.
[40] Aström Eykhoff, "System identification survey." Automatica, vol. 7, no. 2, pp. 123-162, 1971.
[41] W. Yu, Tan, C. K. Liu, and G. Turk, "Preparing for the unknown: Learning universal policy with online system identification." 2017.
[42] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, "Sim-tt-real: Learning agile locomotion for quadruped robots," 2018.
[43] R. Antonova, J. Yang, Sundaresan, D. Fox, Ramos, and J. Bohg, "A bayesian treatment of real-to-sim deformable object manipulation." IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 5819-5826. 2022.
[44] A. Z. Ren, H. Dai, Burchfiel, A. Majumdar, "Adaptsim: Task-driven simulation adaptation for sim-to-real transfer," in Proceedings of the Conference on Robot Learning (CoRL), 2023.
[45] Arjovsky, Chintala, and L. Bottou, "Wasserstein generative adversarial networks." Proceedings of the 34th International Conference on Muchine Learning Volume 70, ser. ICML' 17. JMLR.org, 2017,
2 213-223.
[46] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire,
A. Handa, and G. State. "Isaac gym: High performance gpu-based physics simulation for robot learning." 2021.
[47] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor," 2018.
[48] Н. Fu, K. Tang, Y. Lu, Y. Qi. G. Deng, F. Sung, and C. Chen, "Ess-infoGAIL: Semi-supervised imitation learning from imbalanced demonstrations," seveth Conference Neural Information Processing Systems, 2023. [Online]. Available: http://pwpaeeeeeeeee/o/oottttttd
[49] N. Hansen, "The evolution strategy: A tutorial," arXiv preprint arXin:1604.00072,
13

---

A Appendix
A.1 Proof of Difffusion Evolution
In the diffusion phase, let zi denote the original data point and 2r becomes entirely Gaussian noise. This process can be expressed as:
Vo + 14c,
(12)
where the total noise (0, added to the data zo at time step E [0, is controlled by os, which decreases (gg = to ar ss 0. Meanwhile, a noise prediction module Eg is trained to minimize this loss function in diffusion phase:
E P ||so (Var + v1 a,e, t)
(13)
where Pdota denotes the distribution of the training data. Under the DDIM framework 25 the denoising phase is defined as:
va--1
81
( ()
+ V1 ar-1 ofco(Es, t) + oww
(14)
Var
where o, controls the amount of random perturbations w ~ _N(0, I) added during the denoising phase. Given the diffusion process by Equation 12 versa:
Zo can be expressed by the noise and vise
Et
046 /a
Tt. Va
To
and
(15)
VI a2
by substituting Equation rewritten
15
to Equation to estimate En and é, the denosing phase can be
F--1
04
11-11-1-1 - o72 + aww.
(16)
Since the denoising step in diffusion models requires an estimation of zp. DE derive it from sample x and the corresponding fitness fxx) The estimation of zu can be expressed conditional probability
p(200 = x (.)). Using Bayes' theorem and p(20 = x) glf(x)) yields:
p(aa x/(x)) p(æe|<0 = x)p(xo x)) (()1))99((())
(17)
p(z,)
p(x+)
Here, p(xr|=0 = x) can be computed easily by N vai., 1 - az) given the design diffusion process. Since deep-learning-aase based diffusion models are trained using mean squared error loss, the Tn estimated by F, should the weighted average of the sample æ. Hence, the estimation function of becomes:
Σ glf(x)) N(ri Voue, at)) E, (18)
B,(I,,a,))
Σ
p(2o = 2/21)0
p(xx)
(a))
where Peval is the evaluation sample on which we compute the fitness score, here given by the current - (N), of N individuals. Equation (88 has three weight terms: The first term g (x)) assigns larger weights to high fitness samples. For each individual sample ws, the
population X, = (z/ (1) (2) "?.,,
second Gaussian term N V(x;; /azz, 1 ax) makes each individual only sensitive to local neighbors
of evaluation samples. The third term p(x;) a normalization term. Hence, 20 can be simplified to:
fo o.,
1
Σ
g(s(x))W(ar Voiz, 04)
(19)
7
,EX.
where Z is the normalization term:
Z = p(xx) Σ
glf(x))N(as Vanz, 1 as). zeX.
(20)
When substituting Equation 19) into Equation
15 express noise prediction é as:
F1 v You fo(
Ö(x,, ar t)
(21)
vI 04
14

---

After we successfully estimate the optimal point Zo and the noise é, the next generation of parameters can be generated by using Equation (T6).
Previous black-box optimization methods such as evolution strategies (ES) typically model the parameter distribution as Gaussian, updating its mean and covariance in each sample iteration. However, the new samples are randomly drawn from the updated distribution and lack explicit control over the convergence trajectory, often resulting in slow convergence and low final accuracy. Instead of blindly perturbing parameters in each iteration, DE performs a re-weighted posterior estimation of the optimum based on observed samples. This allows the search process to focus more quickly on regions close to real data distribution.
Specifically, DEAL updates the parameter distribution using the Equation TTP. where the estimated
optimal parameter 0g given by Equation (I.
which can be rewritten follows:
1 Z
E
(0(0,, BE
Σ
olf(0)
Va aa) a, g(I,)
(22)
Ejeny
p(0-=1) 0) p(0:160=0)
p(0, 0n = 0) acts as neighborhood weight around parameter 0t. p(0p 0) represents probability derived by applying a softmax function g(.) to the discriminator's output, reflecting the likelihood that state transition under parameter resembles one from the real world. This formulation effectively uses softmax-style weighting to emphasize high-fitness regions, guiding & rapidly toward the true optimal parameters. As result, the estimate dg serves as a proxy for the gradient direction the WGAN's objective function Equation (3 as follows:
In Equation 22
-Ess,,,s) -dd (ptarget (a) as a(s.as')~a55000 (0,50) D(s, a,s')]
(23)
The diffusion noise schedule also helps stablize the convergence by determining the update step at each iteration. As + O, Ot + and - 0, reducing the variance of N(r;; /o,0, - 04) and encouraging fine-grate fine-grained, local updates in the later stages. The update 0:-1 + &g serves denoising step, similar to the reverse process in diffusion models. With a well-trained discriminator, this process enables stable convergence toward the true optimum.
Additionally, unlike traditional ES that rely on elitism or truncation, which can lose global infor- mation and lead premature convergence to local optima, DEAL utilizes the entire population in Equation 22). It assigns soft weights based on discriminator-derivee direction of the WGAN's objective function. This leads to more targeted and stable updates, offering improved convergence stability, efficiency and accuracy over previous methods.
fitness to estimate the gradient
A.2 Computation cost
The computation time cost presented in Table 3 is based on searches conducted over 50 steps on a PC equipped with an Intel i5-14600KF and an RTX 4060 Ti.
Table Average computation time cost.
Task
DEAL
EASI
BO
Ant Cartpole Go2
I min 38
min 43
7 min 05 S
I min 15 s
min 35 s
6 min 55 s
min 03
5 min 26 s
21 min 17
Humanoid
1 min 43
I min 52 s
42 min 54
AllegroHand
54 min 05 s
54 min 15
125 min 04
A.3 Data budget
All tasks run 200 parallel environments, but the trajectory lengths differ across tasks. Table Table below lists the data budget allocated DEAL and the bascline methods for each task.
15

---

Table Data Budget (200 parallel environments trajectory length).
Data budget
Ant
CartPole
Humanoid
Go2
AllegroHand
For DEAL and baselines
200 200
200 x 200
200 200
200 250
200 600
A.4 Impact of trajectories quality
Current learning-based system identification methods do require the collection of trajectories with a certain quality in the real world. However, even trajectories from very poor policies still encode physical dynamics information. When using the same policy to search parameters, the failed behavior should be reproduced by adjusting those parameters. To address the concern regarding DEAL's reliance trajectory quality, we train the UDR policies for the CartPole and Humanoid in simulation, and take the policies at various training checkpoints to collect a 200-step trajectory in the target environment. We then search for parameters within [ @target 3 x (rorget] to reflect the impact of the policies performance on the search results. the Tables | 50 denotes the policy after 50 training iterations, and the policy only starts converge in the last row.
Table Average search results each checkpoint.
[terations(CartPoee)
Avg. Search Error (%)
[terations(Humaniid)
Avg. Search Error (%)
50
8.84 : 2.96
5e3
13.59 + 5.37
100
7.87 + 2.35
le4
14.70 + 4.80
250
7.19 + 2.96
2e4
13.08 + 5.00
1000
7.34 + 2.88
2.5e4
10.24 + 4.13
2500
5.10 = 2.02
3e4
10.78 + 3.40
Contrary to concerns about its reliance on trajectory quality, DEAL demonstrates strong robust- and still accurately search parameters when using low-quality trajectories collected from early-stage policies. As training progresses and trajectory quality improves, DEAL's search performance improves accordingly.
A.5 Comparison with model-based methods
To assess the improvement of DEAL in finding the optimal parameters comparison with a non- convex optimization problem formulation. We further compare DEAL with several strong model- based baselines in the case of a simple robotic system with non-linear dynamics, including least- squares estimation and Extended Kalman Filter(EKF). In this section, conduct parameter search experiments on the CartPole task using the same trajectory data for all three methods, the results are reported in the Table
Table 6: Comparison with model-based methods.
Avg. Search Error (%)
CartPole
DEAL
6.941.11
Model-based EKF
37.4+8.8
Least-squares
53.0:119.6
As shown in the table, EKF suffers from significant bias due to lincarization errors when estimating nonlinear parameters. The least-squares estimation exhibits relatively large search errors, and the results are highly sensitive to random initialization. In contrast, DEAL demonstrates stable and accurate search performance in such nonlinear systems.
A.6 Broader impacts
In our work, we propose DEAL, a novel framework that leverages diffusion evolution and adversarial learning to align simulated and real-world dynamics with minimal real-world demonstration. By
16

---

narrowing the reality gap, DEAL can accelerate the deployment of RL controllers in robotics, reducing the need for costly and time-consune consuming hardware trials. This has the potential to democratize advanced robotic applications across industries induuries--manufcaturin both development time and resource requirements. At the same time, overreliance on simulator- guided optimization carries the risk that unmodeled real-world complexities sensor unanticipated contacts may still lead to failures during long-term operation. Thus, while DEAL substantially improves sim-to-real transfer. it should be complemented by continued investment in robust robot design, sensor redundancy, and online adaptation schemes, Future work should explore hardware hardware-software co-design to further to address the sim-to-real challenge.
logistics, healthcare by lowering
sensor degradation,
A.7 Physical parameter list
The parameters that are symmetrically distributed on the robot's limbs should be chosen to be the same or similar values to ensure successful training, however, during the parameter search process, they are still treated as independent parameters for searching.
Table 7: Parameter list in the Ant Environment.
Parameter
Target Value
Unit
Contact Friction
1.5
Contact Restitution
0.01
Body-Leg Motor Friction
0.2
Body-Leg Motor Damping
0.3
N m s/rad
Body-Leg Motor Armature
0.1
kg m
Foot-Leg Motor Friction
0.1
Foot-Leg Motor Damping
0.2
N.m m s/rad
Foot-Leg Motor Armature
0.1
kg m²
Foot Mass
0.1
kg
Leg Mass
0.2
kg
Body Mass
1.0
kg
Table 8: Parameter list in the Cartpole Environment.
Parameter
Target Value
Unit
Pole Length
0.3
Pole Mass
0.1
kg
Cart Mass
0.3
kg
Pole DOF Friction
0.1
Pole DOF Dompingg
0.1
N m s/rad kg m²
Pole DOF Amaturee
0.2
Cart DDF_FFrictio
0.1
Cart PID P
0.1
N m-rrad
Cart PID
0.1
N m s/rad
Cart EffortLimit
0.2
N m/s
Cart Vel
1.0
Table 9: Parameter list in the Go2 Environment.
Parameter
Target Value
Unit
Hip Damping 4
20
N m . s/rad
Hip Stiffnessx4
0.5
N m/rad
Calf Damping x4
20
N m s/rad
Calf Stiffnessx4
0.5
N m/rad
Thigh Damping
20
N m s/rad
Thigh StiffnessX4
0.5
N m/rad
17

---

Table 10: Parameter list in the AllegroHand Environment.
Parameter
Target Value
Unit
Contact Frictionx22
1.0
Contact Restitutionx22
0.01
Dof Friction 16
0.01
Dof Dampingx16 16
0.1
N m s/rad
Dof Armature 16
0.001
kg m²
Dof Stiffness 16
3.0
N m/marad
Allegro_Moutd Mount Mass
0.47
kg
Index_ Link___nnk____ 0 Mass
0,012
kg
mndex_Links Mass
0.065
kg
Index_Linke 2 Mass
0.036
kg
Index_Link 3 Mass
0.031
kg
Middle Link 0 Mass
0,012
kg
Middle Link Mass
0.065
kg
Middle_Link
Link 2 Mass
0.036
kg
Middle_I.nne Mass
0.031
kg
Ring_ Ring_Link__0 0 Mass
0.012
kg
Ring_Link: Mass
0.065
kg
King_Links 2 Mass
0.036
kg
Mass
0.031
kg
Thumb_Ling 0 Mass
0.018
kg
Thumb Limnb__ikk_] Mass
0.012
kg
Thumb Link 2 Mass
0.038
kg
Thumb Link 3 Mass
0.060
kg
18

---

Table 11: Parameter list in the Humanoid Environment.
Parameter
Target Value
Unit
Contact Frictionx4
1.0
Contact Restitution
0.01
Abdomen Friction
0.01
Abdomen Y Damping
5.0
N m s/rad
Abdomen Armature
0.02
kg m²
Abdomen Y Stiffiess
20.0
N -m/radd
Abdomen Z Friction
0.01
Abdomen Damping
5.0
N m s/rad kg m² N -mmradd
Abdomen Z Armature
0.01
Abdomen Z Stiffness
20.0
Abdomen X Friction
0.01
Abdomen Damping
5.0
N m s/rad
Abdomen X Armature
0.01
kg m²
Abdomen Stiffness
10.0
N m/rad
Hip_x X Friction x2
0.01
Hip_X Dampingx2
5.0
N m s/rad
Hip_X Armaturex2
0,01
kg m²
Hip_X Stiffness
10.0
N 1-m/rad
Hip Frictionx2
0.01
Hip Z Dampingx2
5.0
N m s/rad
Hip_7. Armaturex2
0.01
kg m²
Hip Z Stiffness x2
10.0
N m/rad
Hip_Y Y Frictionx2
0.01
Hip_Y Dampingx2
5.0
N m s/rad
Hip_Y Armature
0.01
kg m²
Hip_Y Stiffnessx2
20.0
N .-//rad
Knee Frictionx2
0.01
Knee Dampingx2
0.1
N m s/rad
Knee Armature x2
0.007
kg m²
Knee Stiffnessx2
5.0
N N-mrrad
Ankle X Frictionx2
0.01
Ankle_X Dampingx2
1.0
N mmmm s/rad kg m²
Ankle_ Armaturex2
0.006
Ankle_X Stiffnessx2
2.0
N .-m/rad
Ankle Y Frictionx2
0.01
Ankle Y Dampingx2
1.0
N Mmmm s/rad kg m²
Ankle Y Armaturex? x2
0.006
Ankle Stiffness
2.0
N --mraad
Shoulder Friction
0.01
Shoulder Dampingx2
5.0
N m s/rad
Shoulder1 Armaturex2
0.01
kg mi²
Shoulder1 Stiffnessx2
10.0
N m/rad
Shoulder2 Frictionx2
0.01
Shoulder2 Dampingx2
5.0
N m s/rad
Shoulder2 Armaturex2
0.01
kg m²
Shoulder2 Stiffnessx2
10.0
N -m/rad
Elbow Frictionx2
0.01
Elbow Dampingx2
1.0
N m s/rad
Elbow Armaturex2
0.006
kg m²
Elbow Stiffnessx2
2.0
N M-/raad
19

---

A.8 Error bars
These error bars are based on calculations from 95% CI.
Table 12: Average search error percentage(x100%%
Cartpole
Method =
==15
=22
DEAL
0.85:10.06
1.74:10.10
2.63:10.13
DEALAPN
1.90=0.11
2.85=
3.76::0.17
DEALAFN
1.67:0.10
2.96:10.06
4.28:10.12
EASI
4.03+0.50
6.12
8.40::0.4
BO
5.20++0.04
8.52+0016
11.78+0.25
Table 13: Average search error percentage((1100%)
Humanoid
Method DEAL
{ =
1=15
{ =0
0.81+0.02
1.65+00.04
2.50
DEALAPN
2.40=00.06
3.53+0007
4.58
DEALAFN
1.57:0.02
2.79:10.02
4.03:10.03
EASI
2.94
4.64
6.03
BO
4.67+0.01
7.29400.01
10.06
Table 14: Average search error percentagel 100%).
AllegroHand
Method =110
=
=2
DEAL
0.8310.02
1.69
2.57::0.05
DEALAPN
2.60+10.33
3.80=0.05
4.92::0.05
DEALAFN
1.5940.01
2.8010.17
4.06.10.20
EASI 0.12
6.82+10.00
9.32+0.25
BO
=0.01
6.32++0.03
9.16
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the main contributions, including the key method and scope of the research, which supported by the theoretical and experimental results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper includes a dedicated "Limitations" section Section 5 where we
discuss the potential weaknesses of our approach.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: We have provided proofs of the relevant theories used the paper in the main text and Appendix A.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer:[Yor Yes]
Justification: The method proposed in this paper is easy to reproduce, and we introduce detailed information of method implementation and experiment settings in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We in the process of organizing the experiment code and we will provide open access to well-documented code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic//uuddes/Coeeeubbiis cosssiiinolyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppplcccuui.eeaeaecccuea
Stleee//CooSSSbmsssS.Si. www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant training and test details are described in Section and Ap- pendix A.7
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are reported for key results. All figures and tables include erтor bars capturing the variability due different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies the type of compute resources used, including CPU and GPU specifications, etc. These details are provided in Section 4 and Appendix A.2
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics o/ innee
Answer: [Yes]
Justification: The research adheres the NeurlPS Code of Ethics, with careful consideration given to ethical standards throughout the study.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: The potential socictal impacts, both positive and negative, are discussed the Appendix A.6
Guidelines:
24

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The method proposed in this paper is mainly applied to robot control tasks and poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in this research are properly credited, and their licenses and terms of use are respected. This information is provided in the references and acknowledgment sections.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
25

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thoode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets, including datasets and code, are thoroughly documented, and this documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did not use L.MSs as component the method in this study.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
27


Reviews:
[{"review_id": "etFqsyLr2k", "reviewer": "Reviewer_rbd9", "summary": "This paper addresses the problem of  System Identification by introducing a novel method that combines Diffusion Evolution (DE) with Adversarial Learning (AL). The objective is to close the gap between simulator (source) and real-world (target) distributions. The proposed approach follows an iterative process: DE is used as a generator, then a policy trained via UDR is executed in both source and target environments. The discriminator then evaluates the similarity between the two domains using those rollouts. A noise scheduling strategy is introduced to balance exploration and exploitation. The method is evaluated on a wide range of environments, covering both sim-to-sim and sim-to-real tasks, and compared against several baseline approaches.", "questions": "- The paper presents tasks where the DEAL framework outperforms SOTA. But are there cases or scenarios where it underperforms ? \n- Running rollouts on target environment at each iteration may be costly. How would modify your approach to reduce reliance on real-world rollouts ? What happens if only a fixed set of real-world data is available ? Which is often the case in practice\n\nAs I found the main conceptual contribution a bit light, I am assigning a score of 3. But if authors can convincingly address my questions, I would be more than happy to raise my score.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The paper is clearly written and easy to follow\n- Experiments cover both Sim-to-Sim and Sim-to-Real tasks. \n- DEAL appears to outperform SOTA methods in these tasks\n\nWeaknesses\n\n- The idea of computing a similarity measure based on collected rollouts on source and domain to optimize a GAN-style process is not new (eg SimGAN). \n- The main conceptual novelty of this paper is to use a Diffusion Evolution process as a generator, which may be a relatively modest contribution. Beyond empirical performance, is there any theoretical justification for using DE in terms of convergence or sample efficiency ?\n- I did not find detailed description of the discriminator in the paper.  What architecture is used ? How is it regularized to avoid overfitting ?", "was_revised": true, "final_justification": "Authors have addressed most of my concerns, which makes me happy to raise my score to 4", "rebuttal": "`Q1.The idea of computing a similarity measure based on collected rollouts on source and domain to optimize a GAN-style process is not new (eg SimGAN).`\n\n`A1.` We appreciate the inspiration provided by SimGAN. DEAL introduces several key modifications to address known issues in GAN-style system identification.\n\n1. Prior approaches focused solely on comparing the similarity of simulator rollouts to real-world demonstrations, with the single goal of making simulated trajectories mirror real-world demonstrations as closely as possible. In contrast, our approach shifts the focus: we employ a discriminator to assess the similarity of physical state transitions, aiming to fundamentally bridge the gap between simulation and reality. This not only enables more efficient use of real-world data but also leads to improved performance.\n\n2. Previous methods (such as SimGAN) adopt the traditional GAN architecture, using a neural network generator to produce simulator parameters for optimization. However, GAN training is highly unstable, and training the neural generator is time-consuming, requiring significant effort in tuning and debugging. In our work, we introduce Diffusion Evolution (DE) as an alternative for parameter search, which significantly accelerates simulator parameter optimization. Moreover, the combination of DE and the discriminator offers high training stability, avoiding the instability issues commonly encountered in traditional GAN-based methods.\n\n`Q2. Is there any theoretical justification for using DE in terms of convergence or sample efficiency ?`\n\n`A2.` GAN-style parameter search can be viewed as a black-box optimization problem. In DEAL, we choose DE as the generator mainly due to its convergence stability and efficiency, making it more suitable for scaling to complex environments.\n\nPrevious black-box optimization methods such as evolution strategies (ES) typically model the parameter distribution as a Gaussian, updating its mean and covariance in each sample iteration. However, the new samples are randomly drawn from the updated distribution and lack explicit control over the convergence trajectory, often resulting in slow convergence and low final accuracy. **Instead of blindly perturbing parameters in each iteration, DE performs a re-weighted posterior estimation of the optimum based on observed samples.** This allows the search process to focus more quickly on regions close to real data distribution.\n\nSpecifically, DEAL updates the parameter distribution using the following formulation Eq. (9):\n\n$$\n\\theta_{t-1} \\gets \\sqrt{\\alpha_{t-1}}  \\hat \\theta_0 + \\sqrt{1-\\alpha_{t-1}-\\hat\\sigma_t^2}  \\hat\\epsilon +\\hat\\sigma_{t}  \\omega ,\n$$\n\nwhere the estimated optimal parameter $\\hat\\theta_0$ is given by Eq. (1):\n\n$$\n\\hat \\theta_0(\\theta_t,\\alpha,t)=\\frac{1}{Z}\\sum_{\\theta \\in \\Theta_t} \\underbrace{g[f(\\theta)]} _ {p(\\theta_0 = \\theta)} \\underbrace{\\mathcal N(\\theta_t; \\sqrt{\\alpha_t}\\theta, 1-\\alpha_t)} _ {p(\\theta_t | \\theta_0 = \\theta)}\\theta ,  \\quad g(x_i) = \\frac{e^{x_i}}{\\sum_{j} e^{x_j}}.\n$$\n\nIn Eq. (1),  $p(\\theta_t \\mid \\theta_0 = \\theta)$ is defined by the forward diffusion process: $\\theta_t = \\sqrt{\\alpha_t}\\theta_0 + \\sqrt{1-\\alpha_t}\\epsilon$, it acts as a neighborhood weight around parameter $\\theta_t$. $p(\\theta_0 = \\theta)$ represents a probability derived by applying a softmax function $g(\\cdot)$ to the discriminator's output, reflecting the likelihood that a state transition under parameter $\\theta$ resembles one from the real environment. This formulation effectively uses a softmax-style weighting to emphasize high-fitness regions, guiding $\\hat\\theta_0$ rapidly toward the true optimal parameters. As a result, the estimate $\\hat\\theta_0$ serves as a proxy for the gradient direction of the WGAN's objective function $\\mathcal{L}$ in Eq. (5) as follows, then the population is updated directionally via Eq. (9) to move toward this optimal estimate.\n\n$$\n\\underbrace{\\mathcal{L}}_{\\searrow} = \\mathbb{E} _ {(\\mathbf{s},\\mathbf{a},\\mathbf{s}')\\sim d^{\\mathcal{T}}(\\theta^{target},\\pi_0)}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')]-\\underbrace{\\mathbb{E} _ {(\\mathbf{s},\\mathbf{a},\\mathbf{s}')\\sim d^{\\mathcal{S}}(\\theta,\\pi_0)}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')]} _{\\nearrow}.\n$$\n\n**The diffusion noise schedule also helps to stablize the convergence by determining the update step at each iteration.** As $t \\to 0$, $\\alpha_t \\to 1$ and $\\sigma_t \\to 0$, reducing the variance of $\\mathcal{N}(\\theta_t; \\sqrt{\\alpha_t} \\theta, 1 - \\alpha_t)$ and encouraging fine-grained, local updates in the later stages. The update $\\theta_{t-1} \\to \\hat\\theta_0$ serves as a denoising step, similar to the reverse process in diffusion models. With a well-trained discriminator, this process enables stable convergence toward the true optimum.\n\nAdditionally, unlike traditional ES that rely on elitism or truncation, which can lose global information and lead to premature convergence to local optima, **DEAL utilizes the entire population in Eq. (1).** It assigns soft weights based on discriminator-derived fitness to estimate the gradient direction of the WGAN's objective function. This leads to more targeted and stable updates, offering improved convergence stability, efficiency and accuracy over previous methods.\n\nWe appreciate your insightful question regarding the theoretical motivation behind DE. We have incorporated this detailed analysis into the revision to strengthen the conceptual grounding of our method.\n\n`Q3. The detailed description of the discriminator? How is it regularized to avoid overfitting ?`\n\n`A3.` The discriminator used in DEAL adopts **a fully connected MLP with input $(\\mathbf{s},\\mathbf{a},\\mathbf{s}')$, two hidden layers of 256 units with ReLU activation, and a scalar output**. In DEAL, we employs WGAN [1] , which utilizes Wasserstein distance to measure the distribution discrepancy. When two distributions do not overlap, the Wasserstein distance can still accurately reflect their proximity. Furthermore, we apply gradient penalty or weight clipping to regularize the discriminator, preventing overfitting or model collapse.\n\nThank you for pointing out this issue. We have included the detailed description of the discriminator in the revised version.\n\n[1] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. \"Wasserstein generative adversarial networks.\" International conference on machine learning. PMLR, 2017.\n\n`Q4. Are there cases or scenarios where DEAL underperforms ?`\n\n`A4.` In current tasks ranging from low-dimensional to high-dimensional scenarios, DEAL significantly outperforms other baselines. However, DEAL also shares some common issues faced by GAN-style methods.\n\nWhen the target distribution is highly multimodal, existing GAN-style methods may struggle to discriminate among multiple source and target distributions at once, as noted in Appendix A.4 (Limitations). Although diffusion models can capture multimodality,  the discriminator still averages over diverse state transitions rather than converging to distinct solutions. In this case, DEAL tends to homogenize the influence of such multi-source state transitions rather than converge to multiple exact solutions. However, we believe that future work will develop more general and powerful discriminators to match our generator, enabling the simultaneous identification of multiple environment parameter sets.\n\n`Q5. How would modify your approach to reduce reliance on real-world rollouts ? What happens if only a fixed set of real-world data is available ? Which is often the case in practice.`\n\n`A5.` It should be clarified that DEAL uses only offline trajectories. As shown in Alg. 1 and Figure. 1, the policy is not run simultaneously in the simulator and in the real environment during parameter search. In other words, **DEAL itself uses a fixed set of real-world data.** Specifically, we first collect fixed length offline trajectories in the real world using the UDR policy. During parameter search, we only collect trajectories in simulation and sample from the stored real-world trajectories, using the discriminator to distinguish them until their distributions align.\n\nTo further reduce the reliance on real data, we introduce an active learning (AL) module. During UDR policy training, we jointly train an uncertainty predictor, which biases policy actions toward more informative trajectories. In the Ant and CartPole task, we collect one real‑world trajectory of length 200 with each of the standard and active learning models. After conducting parameter search for each, we retrain the new policies and evaluate their returns in the target domain. The table shows the result, which demonstrates that active learning indeed improves sample efficiency by embedding more target‑domain information into trajectories of the same length, thereby improving transfer performance.\n\n| Transfer return | Ant              | CartPole       |\n| --------------- | ---------------- | -------------- |\n| DEAL with AL    | **700.5±29.4** | **183.8±4.1** |\n| DEAL            | 637.3±37.7     | 176.6±4.0     |"}, {"review_id": "GJpeipNGsL", "reviewer": "Reviewer_pNN9", "summary": "RL has demonstrated significant success when trained in simulation, providing cost-efficient and safety advantages. However, when these policies learned in simulation are deployed in the real world, they often experience a reality gap. This paper presents a solution to this problem by utilizing concepts from GANs and diffusion models, introducing a method called Diffusion Evolution with Adversarial Learning (DEAL). Specifically, the paper proposes a new GAN-style framework for system identification, aiming to enhance the simulator and narrow the gap between real-world deployment and simulation. The paper conducted experiments across various tasks, ranging from CartPole (a classic control task) to Humanoid, and showed that the policy learned with the improved simulator dynamics is capable of matching the performance of the policy trained with oracle dynamics.", "questions": "- What is the formula for simulator function that theta is optimizng or apart of? For exmaple, in [1], theta is an extra parameter of the learning dynamics.\n- In the GAN formula in equation 6, what variables in the equation depend on theta?\n- If you are strictly matching the the data distrubiton (s,a,s') from d^t and d^s; how you you decouple the policies being different from the dynamics being different? \n- In DEAL, are you jointly learning a policy, simulator parameters, and a discriminator, and then using the learned dynamics to train a new policy?\n- What are the parameters in the simulators you're trying to learn? How do you decide on these parameters to learn?\n\n\n\n[1] SimGAN: Hybrid Simulator Identification for Domain Adaptation via\nAdversarial Reinforcement Learning", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nReducing the reality gap when training a policy in simulation and deploying it in the real world is very important. The paper's perspective of framing this problem as a GAN and employing modern models to assist in learning the simulator's parameters makes sense. \n\nWeaknesses:\n\nThe paper's writing makes it somewhat challenging to follow. It never clearly defines the simulator parameter theta, leaving it unclear whether this parameter is part of the transition dynamics or something else. The overall concept of the GAN formulation is understandable, but it is not clear how the other parameters relate to theta. While it is obvious how you are optimizing theta using diffusion, it is not clear what exactly is being optimized.", "was_revised": true, "final_justification": "The author's rebuttal addressed my concerns about confusion regarding certain concepts in the paper, and as a result, I raised my score.", "rebuttal": "`Q1. The clear definition of the simulator parameter theta`\n\n`A1.` In this paper, **the specifics of $\\theta$ are detailed in  Section 4.1 and Appendix A.5. It refers to the physical dynamic parameters of the robot or external environment**, such as mass, length and motor coefficients, which directly affect the state transition distribution:  $s_{t+1}\\sim p(s_{t+1}|s_{t},a_{t};\\theta)$. We optimize $\\theta$ in the parameter space by using DE to make the corresponding simulated state transition more similar to the real world state transition, and the discriminator is updated jointly in this process, the objective function Equation (6) can be rewritten as follows:\n\n$$\n\\theta^*=\\mathop{\\arg\\min}\\limits_{\\theta\\in\\mathcal{U}}\\max\\limits_{D}\\mathbb{E}_ {(\\mathbf{s},\\mathbf{a},\\mathbf{s}')\\sim d^{\\mathcal{T}}(\\theta^{target},\\pi_0)}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')]-\\mathbb{E}_{(\\mathbf{s},\\mathbf{a},\\mathbf{s}')\\sim d^{\\mathcal{S}}(\\theta,\\pi_0)}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')],\n$$\n\nwhere $d^{\\mathcal{S}}(\\theta,\\pi_0)$ represents the trajectory collected by $ \\pi_0$ in the simulator parameterized by $\\theta$. In this adversarial process, DE acts as the generator to generate and optimize the $\\theta$, making the simulation trajectory increasingly similar to the real world trajectory  $d^{\\mathcal{T}}(\\theta^{target},\\pi_0)$ from the discriminator's perspective,  while the discriminator aims to distinguish between them as much as possible.\n\nThank you for raising this issue. We have added a clear definition of $\\theta$ in Section 3 of the revised version to facilitate your understanding.\n\n`Q2. What is the formula for simulator function that theta is optimizng or apart of?`\n\n`A2.` $\\theta$ refers to the physical parameters of the robot or external environment. It is not the weight of neural networks or a parameter of function. **The formula for simulator optimization related to $\\theta$ is Equation (6) in answer A1.** In DEAL, we use DE to denoise and optimize $\\theta$ to minimize the discrepancy between the real and simulated state transition distributions as perceived by the discriminator.\n\n`Q3. In the GAN formula in equation 6, what variables in the equation depend on theta?`\n\n`A3.` In Equation (6) from answer A1, **the embedded GAN objective function includes simulated trajectories $d^{\\mathcal{S}}(\\theta,\\pi_0)$, which depend on $\\theta$.** We optimize $\\theta$ using the generator DE and thereby parameterize the simulator, which in turn affects the trajectories collected from the simulator. These trajectories are then used in adversarial training against the discriminator, since the discriminator is always trying to distinguish between simulated and real-world trajectories.\n\n`Q4. How do you decouple the policies being different from the dynamics being different?`\n\n`A4.` **We use the same policy when collecting real-world trajectories and when searching for parameters or collecting simulated trajectories.** This ensures that the parameters we search for are solely aimed at matching the dynamics of the real-world environment.\n\nSpecifically, we first train an initial sub-optimal policy $\\pi_0$ in simulation based on UDR, and then deploy it in the real environment to collect offline trajectories $d^{\\mathcal{T}}(\\theta^{target},\\pi_0)$. During the subsequent parameter search, we continue to use $\\pi_0$ to collect trajectories from the simulator parameterized by $\\theta$, denoted as $d^{\\mathcal{S}}(\\theta,\\pi_0)$. The discriminator then samples from both datasets and assigns scores to distinguish them, updating its network accordingly, while DE updates the distribution of $\\theta$ based on the discriminator's evaluation.\n\n`Q5. In DEAL, are you jointly learning a policy, simulator parameters, and a discriminator?`\n\n`A5.` No, these three components are not trained jointly. The sub-optimal policy $\\pi_0$ is pre-trained via UDR, and its weights are frozen during the parameter search as shown in Figure. 1. Therefore, **only the simulator parameters $\\theta$ and the discriminator are updated during the search process.** This procedure eventually converges to a set of physical parameters that approximate the real-world dynamics, which are then used to reconfigure the simulator and retrain a new policy which better adapted to the real world.\n\n`Q6. What are the parameters in the simulators you're trying to learn? How do you decide on these parameters to learn?`\n\n`A6.` **The parameters contained in $ \\theta$ are listed in Appendix A.5.** These mainly include the parameters that were randomized during the UDR training process, such as the joint PD gains for the Go2 robot or the pole length and mass in the cartpole task. These are key factors affecting sim-to-real transfer, and our goal is to identify more accurate distributions for them, in order to reduce the computational cost and performance degradation introduced by UDR."}, {"review_id": "Z2CLeD3Z4b", "reviewer": "Reviewer_wvcr", "summary": "This paper introduces DEAL (Diffusion Evolution Adversarial Learning), a novel system identification (Sys-Id) framework designed to close the sim-to-real gap for reinforcement learning policies. The core idea is to align the physical dynamics of a simulator with a target real-world environment by accurately identifying the simulator's physical parameters. DEAL employs a dual mechanism: diffusion evolution and adversarial learning. The discriminator's output serves as a fitness score to guide the DE process, which iteratively refines a population of parameter sets to find an optimal configuration that minimizes the discrepancy between simulation and reality. The authors validate DEAL on a range of sim-to-sim and sim-to-real robotics tasks, demonstrating its superior performance in high-dimensional parameter identification and sim-to-real transfer with limited real-world data compared to several baselines.", "questions": "1. How does the framework handle situations where the real-world dynamics are non-stationary? Could DEAL be adapted for online or continuous system identification?\n2. Did you encounter common stability issues in Eq. (6) with UDR?", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n* The primary strength of this work is its demonstrated ability to successfully perform system identification in high-dimensional parameter spaces.\n* The method is explicitly designed to work with only a small amount of few-shot real-world data. The experiments confirm that DEAL can achieve high identification accuracy with sparse trajectories.\n* The combination of Diffusion Evolution and adversarial learning appears to be an appealing approach to the parameter searching for reinforcement learning controllers.\n\n## Weaknesses \n* **Novelty.** While the combination of the components is novel, the underlying techniques are not. The paper leverages existing concepts, and the core contribution is the integration of these parts into a new framework for a specific problem, rather than a fundamental algorithmic breakthrough. \n* **Computational Complexity.** The overall framework still requires iteratively running a large number of parallel simulations (pp. 7), training a discriminator network, and executing the evolutionary search. Although the authors state the search itself is fast, the total computational cost of the pipeline (including initial policy training, data collection, and final policy retraining) could be substantial, potentially limiting its accessibility.\n* The method's success hinges on the quality of the initial real-world trajectories used to train the discriminator. These trajectories are collected using a policy trained with uniform domain randomization. If this policy performs extremely poorly and fails to exhibit informative behaviors in the real world, it may not provide a strong enough signal for the discriminator, potentially limiting the effectiveness", "was_revised": true, "final_justification": "After carefully reading the rebuttal, I have decided to maintain my original score.", "rebuttal": "`Q1. Computational Complexity.` \n\n`A1.` The training cost of the policy itself is not included in our method. Instead, by searching for parameters using DEAL, **the need for manual tuning and policy training time is significantly reduced compared to UDR,** because after parameter search, the domain knowledge that the policy needs to acquire is of high precision and quality. Moreover, if the new parameters remain within the original bounds, we only need to fine-tune the policy.\n\n**Data‑collection cost is also minimal in DEAL.** As described in Section 4.4's Go2 Sim2Real experiment, only a single real robot was run for 20 seconds to collect a trajectory containing 1,000 state transitions for parameter search. Furthermore, if DEAL is only used to search for static parameters, such as mass or motor properties, which are intrinsic physical parameters of the robot, the identified parameters can be reused across different controllers and environments, leading to substantial savings in both computational and time costs.\n\n`Q2. The method's success hinges on the quality of the initial real-world trajectories used to train the discriminator. `\n\n`A2.`  Current learning-based system identification methods do require the collection of trajectories with a certain quality in the real world. However, we believe that even trajectories from very poor policies still encode physical dynamics information. When using the same policy to search parameters, the failed behavior should be reproduced by adjusting those parameters.\n\nTo verify this, we train the  UDR policies for the CartPole and Humanoid in simulation, and take the policies at various training checkpoints to collect a 200-step trajectory in the target environment. We then search for parameters within $[\\tfrac{1}{3}\\times\\theta^{target},3\\times\\theta^{target}]$ to reflect the impact of the policies performance on the search results. In the tables, 50 denotes the policy after 50 training iterations, and the policy only starts to converge in the last row.\n\n| Iterations（CartPole） | Avg. Search Error (%) |\n| ---------------------- | --------------------- |\n| 50                     | 8.84 ± 2.96          |\n| 100                    | 7.87 ± 2.35          |\n| 250                    | 7.19 ± 2.96          |\n| 1000                   | 7.34 ± 2.88          |\n| 2500                   | 5.10 ± 2.02          |\n\n| Iterations（Humanoid） | Avg. Search Error (%) |\n| ---------------------- | --------------------- |\n| 5e3                    | 13.59 ± 5.37         |\n| 1e4                    | 14.70 ± 4.80         |\n| 2e4                    | 13.08 ± 5.00         |\n| 2.5e4                  | 10.24 ± 4.13         |\n| 3e4                    | 10.78 ± 3.40         |\n\n**The tables demonstrate that DEAL is capable of identifying reasonably accurate parameters, even when starting from a poorly performing policy.**  Naturally, as the controller's performance improves, the quality of the generated trajectories increases significantly, leading to search results that more closely match the ground truth.\n\n`Q3. How does the framework handle situations where the real-world dynamics are non-stationary? Could DEAL be adapted for online or continuous system identification?`\n\n`A3.`\n\n1. In non-stationary environments, physical parameters may continuously change, and the collected samples will be multimodal. Since the discriminator can only perform binary classification, it cannot generate effective signals for DE in such cases. However, we believe that future work will develop more general discriminators to match our generator, enabling the accurate identification of non-stationary environment.\n2. In online system identification, parameters found by DEAL can directly boost a model-based controller. But for model-free methods like reinforcement learning, fine-tuning in the updated simulator is needed, which isn't real-time. Therefore, all comparisons in this paper are based on offline parameter search. However, for underactuated mobile robots, we usually focus on identifying key internal physical parameters (e.g., joint motor coefficients and mass) that rarely change significantly and are crucial for sim2real transfer. To better adapt to changes in the external environment, we use the RMA controller with an adaptation module to handle environmental variations. This approach complements DEAL's parameter search and significantly improves transfer performance.\n\n`Q4. Did you encounter common stability issues in Eq. (6) with UDR?`\n\n`A4.` When training vanilla GANs, we encountered stability issues. WGAN[1] has proved that vanilla GANs minimize the JS divergence between distributions, which becomes constant when distributions have no overlap, causing gradient vanishing and unstable training. To address this, we switched to WGAN, which uses the Wasserstein distance to measure distribution gaps. Unlike JS divergence, Wasserstein distance remains effective even when distributions don't overlap. We also applied gradient penalty or weight clipping to enforce Lipschitz continuity and ensure valid Wasserstein distance computation.\n\n[1] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. \"Wasserstein generative adversarial networks.\" International conference on machine learning. PMLR, 2017."}, {"review_id": "9wJjByt4PP", "reviewer": "Reviewer_oQUS", "summary": "This work addresses the problem of system identification for Sim-to-Sim and Sim-to-Real transfer. The authors propose a novel method, DEAL, building upon Diffusion Evolution and Generative Adversarial Networks (GANs). A discriminator is trained to distinguish between two distributions of system parameters: a prior distribution and a target distribution. System identification is then framed as the optimization problem of finding parameter values that fool the trained discriminator. Diffusion Evolution is used to solve this optimization, forming the basis of the DEAL algorithm: a sampling-based method in which the discriminator provides a fitness signal guiding the samples toward high-likelihood regions. An adaptive sampling strategy is introduced, tuning the exploration noise based on fitness. The method is evaluated on several reinforcement learning environments, first through parameter estimation error in Sim-to-Sim setups, then on downstream control tasks in Sim-to-Sim and Sim-to-Real settings. DEAL is compared to several baselines and outperforms them.", "questions": "1. In Figure 4, how can DEAL outperform the oracle? Does the oracle use the true parameters with a suboptimal policy?\n2. Could the authors elaborate on the benefits of their deep-learning-based approach compared to traditional system identification algorithms?\n3. What are the computational and data requirements of DEAL? How do they compare with those of the baselines?\n4. What is the architecture of the discriminator network?\n5. Did the authors consider using active learning during data collection to improve sample efficiency?", "limitations": "The limitations are discussed in the appendix, but would deserve a dedicated paragraph in the main body. See my suggestions in the 'Strength and Weaknesses' section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Globally, the idea of the submission is original and relevant. However, the paper suffers from several clarity issues, particularly in the formulation of the problem and the description of the methodology. This makes it difficult to fully assess the contributions and how they improve upon classical baselines.\n\n### Strengths\n\n- Modeling system identification as a generative modeling problem is an interesting and promising direction.\n- The method is evaluated across several control environments, with results shown both in terms of parameter estimation and downstream control performance.\n- The experimental results are encouraging.\n\n### Weaknesses\n\n**Problem formulation**\n\nThe mathematical problem is not properly introduced, and the paper lacks rigor in its formalism.\n\nThe paper would benefit from a section that clearly formulates the learning problem, including the objective function the proposed method is designed to optimize, independent of a trained discriminator. Typically, system identification aims to minimize some estimation error (e.g., between true and predicted parameters) or improve control performance. Such a formulation would clarify the purpose of the approach and how it differs from the baselines.\n\n**Method structure**\n\nSection 3 lacks a clear structure and separation between the different phases of the method. Several important points remain ambiguous:\n\n- How is the discriminator trained and update? From (5), Algorithm 1., and the experiments, it seems to be trained without a generator. Is the source distribution sampled by running simulations with parameters drawn uniformly?\n- Equation (6) is confusing. Does it define an optimization over fixed discriminator weights, or are these updated jointly with sampling? The similarity to (4) also makes the role of $\\theta$ unclear, as it denotes physical parameters while such a formulation is typically used  to denote neural network weights for GANs.\n- The statement in lines 46–47 that Diffusion Evolution has a \"dual role\" is unclear. In what sense does it serve as both a generator and an optimizer?\n- The link between discriminator training, the objective in (6), and the final DEAL algorithm (Algorithm 1) is not clearly described.\n\n**Evaluation**\n\nThe evaluation process of the proposed algorithm should be stated more thoroughly. It would be clearer if the evaluation metrics were clearly stated in terms of an error function of the true parameters, both for evaluation in parameter space and for evaluation on downstream control tasks.\n\n\n**Computational cost and data efficiency**\n\nThe method’s data and computational efficiency are unclear. In particular:\n\n- DEAL requires training a discriminator on samples from the real-world environment. Training such a network typically requires a substantial amount of data, especially if training happens at each iteration of a search algorithm as in Algorithm 1.\n- Section 4.3 is particularly confusing: what does a demonstration of length 1 mean? How is the discriminator trained in such a low-data regime?\n- The introduction claims that DEAL \"avoids the need for costly real-world data,\" yet the method depends on real samples to train the discriminator. This contradiction should be addressed.\n- Computational cost and sample complexity are key in Sim-to-Real applications. In addition to Table 3, the paper should report training times and simulation budget for DEAL and the baselines, and discuss any trade-offs. The limitations of the proposed approach in direction should be discussed in the main body of the paper.\n\n**Conceptual justification**\n\nThe use of diffusion models for system identification is an original idea, but the motivation is weakly supported:\n\n- Why aren’t classical methods such as least squares regression on physical parameters considered? With access to a simulator, least squares or maximum likelihood estimation seems natural.\n- Is the method meant to work in a model-free setting? If so, this should be clearly stated and contrasted with model-based approaches.\n- More generally, modeling system identification as an adversarial sampling problem needs better theoretical and practical justification. Equation (6) should be compared and contrasted with more standard formulations in the literature.\n\n**Minor clarity issues**\n\n- In line 169, parameters are described as Gaussian-distributed, but iterative sampling algorithms such as diffusion evolution  are usually designed to represent complex, non-Gaussian distributions. This should be clarified.\n- Equation (8) is referenced before it is introduced.\n- There are several typos throughout the paper that hinder readability.", "was_revised": true, "final_justification": "The authors clarified the mathematical formalism in the rebuttal, and addressed my concerns regarding traditional system identification techniques.\n\nI still believe that there is room for improvement in the experimental evaluation.", "rebuttal": "`W1 Problem formulation`\n\n`A1.`We appreciate your feedback and would like to clarify the process of our method for you. \n\nThe goal of DEAL is to minimize the discrepancy between state transition distributions from different domains as follows:\n$$\n\\mathop{\\min}\\limits_{\\theta\\in\\mathcal{U}}\\Vert\\mathcal{P}_t(\\theta^{target}),\\mathcal{P}_s(\\theta)\\Vert.\n$$\n\nOur framework is based on GANs, where the generator and discriminator are trained jointly in an adversarial manner. Accordingly, DEAL also aims to minimize the discrepancy through the joint training of generator and  discriminator. Notably, our generator DE does not involve neural networks, it generates and optimizes the physical parameter distribution $\\theta$ to participate in adversarial training with the discriminator.\n\nTherefore, **$\\theta$ and the discriminator are optimized jointly, rather than training the discriminator first and then optimizing $\\theta$.** To improve clarity, we can express the trajectory generated in the simulator as $d^{\\mathcal{S}}(\\theta,\\pi_0)$ and the trajectory pre-collected in the real world as  $d^{\\mathcal{T}}(\\theta^{target},\\pi_0)$, then the objective function Equation (6) can be written as follows:\n\n$$\n\\theta^*=\\mathop{\\arg\\min}\\limits_{\\theta\\in\\mathcal{U}}\\max\\limits_{D}\\mathbb{E}_ {(\\mathbf{s},\\mathbf{a},\\mathbf{s}')\\sim d^{\\mathcal{T}}(\\theta^{target},\\pi_0)}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')]-\\mathbb{E}_{(\\mathbf{s},\\mathbf{a},\\mathbf{s}')\\sim d^{\\mathcal{S}}(\\theta,\\pi_0)}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')] .\n$$\n\nThis formulation makes it clear that the overall objective in Equation (6) describes the joint optimization of $\\theta$ and the discriminator, the trajectories from both domains are jointly sampled and input to the discriminator for scoring. \n\nSince we are optimizing over the physical parameter distribution $\\theta$ rather than a generator neural network, this may lead to misunderstanding. \nThank you for your question and we have refined the problem formulation in the revised version.\n\n`W2 Method structure`\n\n`A2.`\n\n1. DEAL follows the same paradigm as GANs, where **the discriminator is not pre-trained separately but co-optimized with the generator in an interactive manner.** In Equation (5), the source-domain data $d^{\\mathcal{S}}$ depends on physical parameters $\\theta$ produced by the generator. The inner loop of the discriminator in Algorithm 1 represents epoch-wise training rather than a full pre-training stage. During this interactive process, the source-domain parameters are freshly generated by DE at each step.\n2. **Equation (6) formalizes the joint adversarial optimization of the discriminator and parameter distribution given by DE, rather than optimizing over a fixed, pre-trained discriminator.** In contrast to Equation (4), where the generator learns neural network weights, DEAL performs optimization over a distribution of physical parameters that influence the trajectory, since DE does not involve any neural networks internally.\n3. DE plays a dual role as a diffusion-based generator that produces parameters to configure the simulator while simultaneously optimizing these parameters based on the discriminator's feedback.\n4. To summarize, DEAL does not include separate discriminator pre-training. Equation (6) defines the overall objective of DEAL. The discriminator's objective is embedded in Equation (6), DE generates and optimizes parameters to adversarially minimize this state-transition distribution gap.\n\n`W3 Evaluation`\n\n`A3.`The evaluation metric in Sections 4.1 and 4.3 is the average percentage error of the searched parameters relative to the true parameters, defined as follows:\n$$\n\\epsilon_p = \\mathbb{E}\\left(\\left|\\frac{{\\theta} - \\theta^{target}}{\\theta^{target}}\\right|\\right) \\times 100\\\\%\n$$\nThank you for your suggestion. We have included clear evaluation formulas in the revised version.\n\n`W4 Computational cost and data efficiency / Q3`\n\n`A4.`\n\n1. As DEAL does not require a pre-trained discriminator at each iteration. The discriminator and parameter distribution $\\theta$ are updated jointly, so while some trajectory data is needed, the amount is not excessive.\n2. In Section 4.3, the humanoid task performs sim2sim parameter search using trajectories collected from 200 parallel environments, as mentioned in Experiment Details. Therefore, the real trajectory here refers to 200 = (parallel number = 200)$\\times$(trajectory length = 1) state transitions in total.\n3. In line 52, this phase is followed by \"to train a noise prediction module\", indicating it only refers to not using real samples for training the noise predictor. Since DE simulates a denoising process without involving a diffusion process to train the noise predictor.\n4. Because DEAL does not separately pretrain the discriminator, the time reported in Table 3 represents the combined training cost of both the parameter distribution and the discriminator.\n\nAdditionally, we provide the simulation data budgets for each task, all tasks run 200 parallel environments, but the trajectory lengths differ across tasks. The detailed numbers are shown in the table below.\n\n|Data budget|Ant|CartPole|Humanoid|Go2|AllegroHand|\n|-|-|-|-|-|-|\n|For DEAL and baselines|200*200|200*200|200*200|200*250|200*600|\n\nTo compare the data requirements of DEAL and other baselines, we collect between $\\eta$=1 and $\\eta$=200 trajectories and do parameter search in Humanoid task, then we measure the average search error to evaluate each method's data efficiency.\n\n|Avg. Search Error (%)|$\\eta$ = 1|$\\eta$ = 5|$\\eta$ = 10|$\\eta$ = 50|$\\eta$ = 200|CPU Memory(G)|GPU Memory(G)|\n|-|-|-|-|-|-|-|-|\n|DEAL|12.5±4.4|13.8±5.0|13.6±5.2|13.9±5.1|12.9±4.7|2.3|0.7|\n|EASI|66.0±20.2|54.1±16.4|57.8±20.2|54.2±19.0|58.2±19.4|2.4|0.9|\n|BO|108.5±36.9|107.8±36.7|107.6±37.0|102.5±36.6|97.7±37.1|2.4|0.9|\n\nThe table above shows that **DEAL achieves the best optimization results using the same amount of trajectory, confirming that DEAL requires less data than other baselines.**\n\n`W5 Conceptual justification`\n\n`A5.`\n1. Due to the non-differentiability of simulators, using least squares or maximum likelihood to compute trajectory gradients doesn't allow backpropagation to physical parameters. Moreover, as the parameter space grows exponentially, gradient-based optimization becomes increasingly intractable. In contrast, DEAL leverages the high-dimensional adaptability of diffusion models to scale more efficiently.\n2. We compare DEAL with Model-based method Extended Kalman Filter(EKF) on the CartPole task using the same trajectory data. As shown in the table, EKF suffers from significant bias due to linearization errors when estimating nonlinear parameters.\n|Avg. Search Error (%)|CartPole|\n|-|-|\n|DEAL|6.9±1.1|\n|EKF|37.4±8.8|\n3. Thank you for your suggestion. We have refined and clarified Equation (6).\n\n`W6 Minor clarity issues`\n\n`A6.`In DEAL, population individuals iteratively evolve from initial noise, forming a complex distribution that ultimately converges near target parameters. For improved sampling, we fit a Gaussian to the final population's mean and variance, using it as the retraining distribution.\n\nWe appreciate your feedback and have addressed other clarity issues in the revised version.\n\n`Q1. In Figure 4, how can DEAL outperform the oracle?`\n\n`A7.`When the target parameters are not sufficiently well set, training in the target domain can become relatively more difficult. For example, in tasks like Ant or AllegroHand, excessive mass or weak actuators may slow down oracle training. The searched parameter distribution, due to its bias, may sample easier parameters during training, which can accelerate learning and temporarily result in higher rewards than the oracle. Such fluctuations tend to diminish as training converges, without affecting the oracle’s theoretical optimality.\n\n`Q2. Could the authors elaborate on the benefits of their deep-learning-based approach compared to traditional system identification algorithms?`\n\n`A8.`DEAL does not rely on linear assumptions or explicit gradient access often required by traditional methods. Instead, it performs sample-based blackbox optimization, making it broadly applicable to nonlinear systems. Unlike traditional methods that require covariance estimation in high-dimensional spaces, DEAL avoids numerical instability by directly executing denoising updates via Eq. (9). Through discriminator-guided reweighting in Eq. (1), it efficiently identifies high-fitness regions, effectively leading to more targeted and stable updates. The diffusion process further enables adaptive step size control, allowing DEAL to stably and rapidly converge toward the optimal solution.\n\n`Q4. What is the architecture of the discriminator network?`\n\n`A9.`In this work, the discriminator is implemented as a fully connected MLP with input $(\\mathbf{s},\\mathbf{a},\\mathbf{s}')$, two hidden layers of 256 units with ReLU activation, and a scalar output. Furthermore, We adopt the WGAN framework to address instability during discriminator training.\n\n`Q5. Did the authors consider using active learning during data collection to improve sample efficiency?`\n\n`A10.`We incorporate active learning (AL) for data collection in both Ant and CartPole tasks, one UDR policy is trained with AL by jointly learning an uncertainty predictor to guide exploration toward more informative trajectories, while the other is trained without AL. Each policy is then used to collect a single trajectory of length 200 for parameter search. After retraining new policies with the respective optimized parameters and deploying them in the target domain, we compare their transfer return. As shown in the table below, the results indicate that active learning improves sample efficiency by embedding more target-domain information into trajectories of the same length, thereby enhancing policy adaptability.\n|Transfer return|Ant|CartPole|\n|-|-|-|\n|DEAL with AL|700.5±29.4|183.8±4.1|\n|DEAL|637.3±37.7|176.6±4.0|"}]

Author rebuttal:

Meta-review: This paper proposes DEAL, a system identification method that combines diffusion evolution with adversarial learning for better sim2real transfer in reinforcement learning. By iteratively refining physical parameters using a discriminator for transition similarity and diffusion evolution for parameter optimization, DEAL achieves higher stability and accuracy in high-dimensional settings. Experiments show it outperforms baselines in sim2real transfer while requiring minimal real-world data. Overall, the results are interesting and have sufficient data to back them up. Please consider some of the presentation details raised by the reviewers (adding more details about the discriminator and training of parameters, policies) in a revision of your draft.
Decision: Accept (poster)