EASI: Evolutionary Adversarial Simulator Identification for Sim-to-Real Transfer
Authors: Haoyu Dong
Huiqiao Fu
Wentao Xu
Zhehao Zhou
Chunlin Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Evolutionary adversarial simulator identification
reinforcement learning
sim-to-real transfer
Primary area: reinforcement_learning
TL;DR: We introduce a novel approach of Evolutionary Adversarial Simulator Identification (EASI) by combining Generative Adversarial Network (GAN) and Evolutionary Strategy (ES) to address sim-to-real challenges.

Abstract: Reinforcement Learning (RL) controllers have demonstrated remarkable performance in complex robot control tasks. However, the presence of reality gap often leads to poor performance when deploying policies trained in simulation directly onto real robots. Previous sim-to-real algorithms like Domain Randomization (DR) requires domain-specific expertise and suffers from issues such as reduced control performance and high training costs. In this work, we introduce Evolutionary Adversarial Simulator Identification (EASI), a novel approach that combines Generative Adversarial Network (GAN) and Evolutionary Strategy (ES) to address sim-to-real challenges. Specifically, we consider the problem of sim-to-real as a search problem, where ES acts as a generator in adversarial competition with a neural network discriminator, aiming to find physical parameter distributions that make the state transitions between simulation and reality as similar as possible. The discriminator serves as the fitness function, guiding the evolution of the physical parameter distributions. EASI features simplicity, low cost, and high fidelity, enabling the construction of a more realistic simulator with minimal requirements for real-world data, thus aiding in transferring simulated-trained policies to the real world. We demonstrate the performance of EASI in both sim-to-sim and sim-to-real tasks, showing superior performance compared to existing sim-to-real algorithms.

EASI: Evolutionary Adversarial Simulator Identification for Sim-to-Real Transfer
Haoyu Dong, Huiqiao Fu, Wentao Xu, Zhehao Zhou, Chunlin Chen" Department of Control Science and Intelligent Engineering, School of Management and Engineering, Nanjing University, China [haoyudong.hqfu, wtxu, zhzhou) @ seaail.jju.edu.cm clchen@nju.edu.cc
Abstract
Reinforcement Learning (RL) controllers have demonstrated remarkable perfor- mance in complex robot control tasks. However, the presence of reality gap often leads to poor performance when deploying policies trained in simulation directly onto real robots. Previous sim-to-real algorithms like Domain Randomization (DR) requires -omain--pecific expertise and suffers from issues such as reduced control performance and high training costs. In this work, we introduce Evolution- ary Adversarial Simulator Identification (EASI), novel approach that combines Generative Adversarial Network (GAN) and Evolutionary Strategy (ES) to address sim-to-real challenges. Specifically, we consider problem sim-to-real as a search problem, where ES acts as generator in adversarial competition with neural network discriminator, aiming to find physical parameter distributions that make the state transitions between simulation and reality similar possible. The discriminator serves as the fitness function, guiding the evolution the physi- cal parameter distributions. EASI features simplicity, low cost, and high fidelity, more realistic simulator with minimal requirements
enabling the construction for real-world data, thus aiding in transferring simulated-traine trained policies the real world. We demonstrate the performance of EASI in both sim-to-sim and sim-to-real tasks, showing superior performance compared to existing sim-to-real algorithms.
Introduction
With the development of Reinforcement Learning (RL). an increasing number of RL controllers have demonstrated remarkable performance in complex, high-dimensional robot control tasks (2] Such RL algorithms require minimal domain-spedte knowledge, offering superior control performance and more natural robot locomotion compared to traditional control methods. However, training these RI controllers requires plenty of trial and error processes, it's unsafe and costly to complete training tasks in the real world. On the other hand, simulation provides cheaper and safer way to obtain vast amount of trial data which is more suitable for RL training. Yet, due to simulation inaccuracies, disparities between simulation and reality inevitably exist and simulated-traines policies transferring directly to the real environment often result in poor performance, which is also known as 'reality gap'.
To bridge the gap between simulation and reality, many studies focus sim-to-real transfers to translate simulated-trrs -uuinee policy reality. As one the most popular sim-to-real methods, Domain Randomization (DR) (316] injects random noise into observations, actions, and physical parameters during simulated training, which helps agents adapt different environments, making the policies
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

more robust. However, DR requires a good understanding of the specific domain to determine the appropriate ranges for these random variations. these ranges are set inappropriately, it can hinder the successful transfer of policies. Additionally, DR often needs longer training time to ensure adaptability of the policy to various environments, and always results in suboptimal and conservative performance [6] To tackle these challenges, recent attempts have been made to use real-world data for fostering policy transfer to real-world applications. One approach is optimizing simulator parameters to make the simulation more realistic [7-14]. Unfortunately, these methods still face challenges, such as the need to iteratively collect large amounts of high-cost real-world data, difficulty in applying to high-dimensional control tasks, and instability in the parameter optimization process.
In this work, we introduce Evolutionary Adversarial Simulator Identification (EASI), novel sim-to real method that frames sim-to-rea transfer as a search problem and employs unique collaboration between Generative Adversarial Network (GAN) and Evolutionary Strategy (ES). EASI searches for physical parameters to align state transitions between simulation and reality as closely as possible, ensuring that policies trained in simulation seamlessly transfer to the real world. Within EASI, ES acts as a generator in an adversarial competition with neural network discriminator, which distinguishes state transitions originating from either simulation or reality. The discriminator assigns higher scores to state transitions more likely sampled from reality, acting as the fitness function for ES and guiding the evolution of physical parameter distributions. Concurrently, the discriminator's s ability differentiate between simulated and real state transitions improves through iterative training. EASI features simplicity, low cost, and high fidelity, requiring only a small amount of real-world data to bridge the gap between simulation and reality.
In the experiments, we test EASI on sim-to-sim tasks (Go2, Cartpole, Ant, Ballbalance) and sim-to-real tasks (Go2, Cartpole). We first use Uniform Domain Randomization (UDR) to train a rough initial policy over a wide parameter range, allowing the agent to collect data in the target environment controlled by the initial policy. Then, we employ EASI for simulator identification that makes the simulator most similar to the target domain. The final policy is trained based on these optimized simulator parameters. Our experiments demonstrate that EASI can stably and rapidly search for simulator parameters, and the policies trained within the optimized parameter distribution effectively transfer the target domain. Video and code are shown at our page. In particular, make three main contributions in this work:
1. We propose a novel method, EASI, which employs a unique collaboration between GAN and ES to address sim-to-real challenges.
2. We model sim-to-real transfer as a search problem, searching for simulator parameters in an adversarial process to make state transitions between simulation and reality most similar.
3. We demonstrate the performance of EASI in both sim-to-sim and sim-to-real tasks, showing superior performance compared to existing sim-to-real algorithms.
2 Related Work
Transferring ionulated-trainee policies to the real world in a stable and cost-effective manner has long been a goal for sim-to-real transfers. Previous research has approached sim-to-real transfer from both zero-shot and few-shot perspectives. perspectives.Zero-eee Zero-shot sim-to-real transfer directly trains a policy capable of operating in reality without using any real-world data. Domain Randomization (DR) is one of the most intuitive and widely applied zero-shot sim-to-real methods noise into state observations, actions, and physical parameters to enhance the robustness of the policy, allowing the agent to adapt to various environments. However, DR requires specific domain prior knowledge and mand-ennine to determine the distribution of injected random noise, while also suffering from reduced control performance and high training costs. To improve DR, Akkaya et al. [5) used task performance as a metric to develop a curriculum that gradually increases the level of randomization. Mehta et al. [16] employed GAN to identify the most informative parts of the randomization parameter space, focusing more attention on these areas. Furthermore, some zero shot methods [17 introduced an adaptation module or estimator to predict robot state latent variables or explicitly estimate privileged information that the robot cannot access in reality, allowing the robot to adapt to different reality environments relying only on limited proprioception.
DR injects random
Few-shot sim-to-red transfer leverages real-world data to facilitate policy transfer. Domain adaptation is one of the approaches that allows policy to generalize to reality. Image adaptation methods (20 23

---

used GAN to transform simulated RGB images in a way that is matched to the reality. Grounded Action Transformation (GAT) 224 suggested action transformation that applying the transformed actions in simulation has the same effects as applying the original actions had on reality. In robotic tasks, domain adaptation methods often require a substantial amount of costly real-world data for effective transfer. Inspired by classic system identification (27]22], some efforts focused on optimizing simulators so that the policies trained within them will better adapt to reality. Bayesian optimization was used to find the posterior distribution of simulator parameters based on real-world observations 13 4 30). Chebotar et al.[8] and Chang et al. [1 treated the trajectory mismatch as the cost function and iteratively updated the simulator parameters. Memmel et al. [ and Marija et al. designed effective world exploration policies to sample more informative trajectories for system identification. TuneNet 77 employed supervised learning to train neural network mapping simulation trajectories to parameter gradients toward the real world, Some approaches 33 used RI. to find proper parameters, treating trajectories RL states and parameters as RI actions. SimGAN 10 employed GAN to distinguish between source and target domain dynamics and enhance the hybrid simulator make it more realistic. However, most current simulation parameter optimization methods still suffer from instability, high cost of reality data collecting and are only suitable for low-dimensional control tasks.
In this work, EASI employs GAN to differentiate between state transitions in simulation and reality, using ES to search for parameters that align the simulation more closely with the real world based on the GAN's ' distinction results. EASI requires lower demands on real-world data while enabling faster and more stable searching of simulator parameters. This helps obtain high-fidelity. low-cost simulators to provide a real world like environment for RI. training.
3 Preliminary
3.1 Domain Randomization
Consider a RL problem defined by the Markov Decision Process (MDP) represented by a tuple (8, A, R, 7, P(E)), where S is the state space, A the action space, S x A R represents the reward function, is the discount factor, P(E) S x A x S + R4 represents the system's 's state- transition probability function under simulator parameters ç. The state transition varies depending on the type of system and its specific parameters. We denote the real-world state transition as Pr((rrett and the simulation state transition as (. In RL process, an agent interacts with the environment
and learns policy (als) to maximize the expected discounted return Ep(r(=)
T-1 y'R (Si, a
where p(T[==) = p(s)) IT= P (S++1 [8,, a.) r(a (s:) represents the likelihood of a trajectory 7 (so, ap, ro. Si...., -T-1, aT-1, 17, ST) under π.. Due the existence of the reality gap. policies trained in simulation often perform poorly when directly transferred to the real environment, and the manifestation of the reality gap in MDP lies in the difference between (Ereat) and P. (Exim) To address this issue, DR samples Eeim E E R and sets the simulator's parameters to Eoim at each simulation training round, allowing the RL-trained policy to adapt to various Pa(Eesm). thus improving system robustness. However, to train policies that perform well in the real world using DR, is necessary to ensure that there exists 5" E E such that (Erraat z P&(E"). which means needs specific domain knowledge to determine =.
3.2 Evolutionary Strategy
Evolutionary Strategy (ES) is a method that mimics biological evolution to solve high-dimensional, continuous-value domain problems. ES algorithm addresses the following search problem: maximize nonlinear fitness function that is a mapping from search space I to fitness value R. In each iteration (generation). the individual from the previous generation undergoes recombination and mutation to reproduce the next generation of individuals. Each offspring is evaluated through the fitness function, and individuals with the higher scores form the next generation's 's population.
In this work, we use the discriminator as the fitness function of ES to evaluate the similarity of state transition, guiding ES to find Esim that deceives the discriminator to minimize the distance between Pr (Ereat) and Pa (Esi)).
3

---

Demonstrations
Evolution Strategy
Euv. parameters
selection
rocomitnation musitinn
Policy
Simulator
Discriminator
simulation riformor
State transitions
Fitness
Figure 1: Schematic overview of EASI. ES acts as generator in adversarial competition with a neural network discriminator distinguishing between simulation and reality state transitions. The dis- criminator serves as the fitness function, guiding the evolution of the physical parameter distributions.
3.3
Generative Adversarial Network
GAN
consists two neural networks, generator G and discriminator D, engaged in an adversarial process to generate new data with the same statistics as the training set. G and D play a two-player minimax game with value function VGG, D), which be described as
34
min max V(G, D) = (ex-paaraa(t (logg ((x)) + EE-p(t)) - DDG(())))) G D
(I)
where p(z) is the prior on input noise variables, Pdata is the training set. G and D continuously adjust parameters, with the ultimate goal of training a generator strong enough to let the discriminator unable to distinguish whether the outputs of the generator network are generated or sampled from training set.
Evolutionary Adversarial Simulator Identification
4.1 Imitate State Transitions
Modeling the processes in reality and simulation MDPs, the primary difference between the two MDPs lies in their state transitions (Sreat) and (S(mii)). In this paper, we focus on minimizing the state transition gap between simulation and reality. For simulation with variable physical parameters, we aim to find the distribution of the parameters achieve
= arg min [PPr Pa Pa(Esim)ll
Esim
In EASI, we discriminator to distinguish whether state transitions come from simulation reality, the discriminator (ss, a, s') can be trained using
max BaM [log D(s, s')) (s llog(1 D(s,
where (MM(s, a, s') and d a, s') denote the distribution of action a causing state transition from state to s' in the demonstration dataset and in simulation respectively. As demonstrated in previous work (34], the discriminator serves as a measure of the and (Essm)).
divergence between (r(Gre
important to acknowledge that, due to inherent design factors in the simulator, achieving state transitions that are precisely identical to those in the real world may be impossible. Nonetheless, our experiments have shown that EAST significantly enhances the simulator's realism, allowing policies trained in simulation to transfer more effectively to real world.

---

Algorithm 1: EASI
Input: Initial parameter distribution = motion control policy To, demonstration state
transition M, simulation state transition buffer B
Output: Parameter distribution =*
1 for generation 0, 2, Go
2 Sample individuals by ((() ()) j 1,,3, N,
for simulation environment j = 1.2,3, , do
Set simulator parameters }
Use To to sample trajectory Tj Store Tj in B
7
end
for update step = 1, 2, ..................... do
B.M sample batch of K state fannitions((s;; ak. a {k)] from M
8 sample batch of K state transitions ((Sk, ag, k== from B
Update D according Equation 4 using b- and b
12
Clip network weights of D
13
end
Calculate discriminator reward r (D,, 1,,,33... ..
Using ES to find next generation distribution E(+ -ES (g(0) r()))
15
16 end
4.2
Wasserstein Loss
The standard GAN discriminator objective detailed in Equation uses a binary cross-entropy loss function which tends encounter the problem of vanishing gradients due saturation regions 351. Specifically, if the discriminator is overly powerful, may excessively classify state transitions, leading to identical bad rewards for all transitions sampled from the simulation. This makes it difficult to identify which parameters yield more realistic state transitions. Additionally, due to inherent discrepancies between the simulator and reality, the state transitions generated by the simulator may not perfectly align with those in the real world. further complicating the challenges posed by a strong discriminator: To address such challenges, WGAN-style discriminator (36) is proposed mitigate the issue of gradient vanishing and has proven effective for robot-related tasks (2). In our work, we employ Wasserstein loss to update the discriminator using
max (BM [(s, s')] (g D(ss, a,s')).
This WGAN-style discriminators is an efficient approximation to the Earth-Mover distance which more effectively and smoothly measures the distance between two probability distributions. We clip the network weights to [-0.01, 0.01] as the original WGAN did to ensure the Lipschitz continuity of D(s, a, s') in the training process. Using Wasserstein loss, could better evaluate the similarity between simulation and real world state transitions.
4.3 Evolutionary Strategy for Parameter Search
Our goal is to find parameter distribution (e.g. Gaussian distribution) that makes the simulator most similar to the demonstration. The distance between P-(Sreat) and P. P.(Ssim) measured by D(s, s') and what we need to do is to find the best parameter distribution E* that get the highest reward D(s, a, s'), which could be described as
Ex arg min max (s. [D(s, a,s') (ss(s.a.s)) (sssssss))
(5)
=
In EASI, we employ the ES as the generator, with goal of creating parameter distribution for the simulator that closely mimics the state transitions observed in the demonstration data. EASI uses the ES to select parameters that receive higher rewards from the discriminator, designating these parameters clites. The elites then undergo recombination and mutation to produce the next
5

---

generation of parameters. The discriminator evaluates whether the state transitions in the generated trajectories are sufficiently similar to those of the expert demonstrations, assigning scores based on the degree of similarity. Simultaneously, the discriminator is trained using the trajectories collected during simulation and demonstration, guided by Equation 4 With the selection process driven by the ES, the state transitions produced by the simulator increasingly resemble those in the demonstration data, and the discriminator can better distinguish the gap between simulated and demonstration state transitions. Ultimately, this algorithm yields set of simulator parameters that align the state transitions the simulated environment with those in the real world. This optimized parameter set then utilized to train policies that be directly applied in real world. The schematic overview of the EASI architecture is shown in Fig. and the pseudo-code is shown in Algorithm II
It's notable that when sampling trajectory from simulation and real world, the motion control policy we use does not necessarily match the final desired policy and the robot may controlled by MPC 33), RI,, or even manually operated by humans, long as it can collect state transition containing enough state transition information we need. This characteristic of our algorithm significantly reduces the cost of collecting real data. With our algorithm, using rough policy run in the real world to collect suboptimal trajectories is enough to find proper parameter for the simulator.
Experiment
In this section, we conduct experiments in various tasks to validate the performance of EASI and answer three questions:
1. Can EASI enhance the simulator's similarity to real-world?
2. Can EASI improve performance in sim-to-real transfer tasks?
3. How much real-world data is required for EASI?
Task Setup We test EASI in 4 sim-to-sim tasks illustrated in Fig. 2 and 2 sim-to-real tasks presented in Fig. 3 (1) Cartpole environment includes an inverse pendulum connected to 1-DoF cart. The goal of Cartpole task is to keep the pendulum on the cart balanced for as long as possible. (2) Go2 environment a 12-DoF Unitree Go2 quadruped robot, with each leg capable of 3-DoF. The goal of Go2 task is to keep the quadruped run forward and track specified speed in natural locomotion. common
(3) Ant environment an 8-DOF quadruped robot consisting of four legs attached base. The goal of Ant task is controlling the ant run as fast as possible. (4) Ballbalance environment consists a table with three legs and a ball placed on the table. Each leg has DoF to control the target angle of the leg joint. The goal of Ballbalance is to stabilize the ball at target position on the table.
We utilize Isaac Gym [39 as the simulator. In the simulator identification process, we create 300 parallel environments for EASI to evaluate the similarity between reality and simulator with different
Figure 2: Experiment tasks in simulation: Cartpole, Go2, Ant, and Ballbalance, presented in order from left to right.
Figure 3: Left: Cartpole task in reality, aiming to keep the pendulum on the cart balanced for as long as possible. Right: Go2 task in reality, aiming keep running forward and track specified speed.

---

parameters. For Caripole, Ant, and Ballbalance, we use SAC 40 to train a policy (4 layer MLP) as the locomotion controller. As for Go2, we employ Ess-InfoGAIL 41 to train quadruped controller.
Baselines (1) Uniform Domain Randomization (UDR): Uniformly sampling parameters from BUDR at the beginning of each training iteration. (2) Oracle: Directly training in the pseudo-real environment in sim-to-sim experiments, representing the ideal upper bound for sim-to-sim transfer tasks. (3) FineTune Fine-tune the UDR policy using real-world data, enabling the UDR to adapt to the real world in advance to improve transfer performance. (4) GARAT: GARAT (6] adjusts actions from the policy such that, after adjustment, the actions applied in the simulator result in state transitions more similar to those in the real demonstration.
EASI Implementation There no specific requirements for the selection of the ES in EASI, mainstream ES such as CMA-ES[42), NES[43) etc. can be used. In this paper, we employ (μ/Pz- X)- ES [4] the generator with the setting μ 150 and X = 300, which means every generation has 300 individuals and 150 elites are chosen to undergo recombination and mutation to generate the next generation. For the collection of the demonstration dataset, we UDR to train rough policy, and then use the policy to control agents in the target domain collecting trajectories.
EASI can quickly accomplish parameter searching tasks. The discriminator can directly evaluate the similarity between simulator and reality by leveraging a single trajectory sampled from the simulator with specific parameter. Furthermore, in Isaac Gym, we can parallelly collect trajectories in hundreds of environments with different parameters which means we could evaluate hundreds of parameters parallelly. In experiment, running PC equipped with Intel i5 55-13000KF and RTX 4060 Ti, EASI completed the evolutionary adversarial searching process in less than 10 minutes.
5.1 Sim-to-Sim Policy Transfer
To validate the theoretical capability of EASI for simulator identification, we conduct sim- to-sim experiments in tasks (Cartpole, Go2, Ant, and Ballbalance). Initially, set of pa- rameters & selected as the target parame- ters for the simulator, which is regarded as the pseudo-real environment. For UDR, we use SyDR = U x & & x &r] as the uniform pa- rameter distribution and train an initial policy To with UDR. Depending
Go2
Go2
120
50
1.5
50-
Thiph
3.0
6.5
20 Generation
20 Ganaration
Ant
Ant
the difficulty of the task, we set = 3 for the Cartpole, Ant, and Ballbalance tasks, and k- = 2 for Go2 task.
Ruge Rictoo
000
WD
Dof Meteo
1.00
000
WD
0.75
0.50
Then xp is used to collect 200 trajectories in the pseudo-real environment as demonstration state transition M. During the EASI parameter optimization process, two approaches are used for selecting the initial parameter distribution: Within Distribution (WD) and Out of Distribu- tion (OOD). In the WD experiment, the initial parameter distribution =(0) is consistent with the domain randomization distribution EEDn. and the target parameter &, is included within the initial parameter distribution E(O). In the OOD experiment, we deliberately set incorrect initial parameter distribution E()) (e.g. set as U S.] or Ule,. 2 x {r)) so that the target parameter is outside the initial parameter distribution. Taking Go2 and Ant environment as examples, there are parameters for Go2 and 11 parameters for Ant to identify. Here we choose 2 of the parameters in each task to demonstrate the generating process of the parameter. Fig. 4 represents the convergence process of parameters, where the x-axis represents the generations of ES, and the y-axis represents the parameter values. It can be observed from the figure that at the beginning, parameters are uniformly distributed within E(") As the parameter evolution progresses, the parameter distribution quickly adjusts to the vicinity of the target parameters. This experiment showeases EASI's capability to adjust parameter distributions, even when the initial
0.00
20
Generation
Figure 4: Evolution process of parameters. The red dashed line represents the target parameter. WD means target parameter within initial parameter distribution, and OOD means target parameter out of initial parameter distribution.
7

---

Cartpole
Go2
0.2
Neal Emo Naturn
Weal Eit Retum
0.4
Gracie
CASI
0.2
Drace UDA
FASI
UDX
0.00
0.25
0.50
0:75
1.40
1.25
3,50
1.75
Pilicy Snaps
1c3
Parlcy Stopa
1c3
Ant
Ballbalance
300
Beal Ene
220
Nea
100
200
1e4
1c4
Figure 5: Performance on the pseudo-real environment over the process of training.
distribution is incorrect. This implies that by specifying rough initial parameter range, EASI can effectively align the parameter distribution within the desired range.
After parameter optimization, we utilize the con- verged parameter distribution =" train policies. In each task, EASI runs three times with different random seeds to avoid random- ness. Figure depicts the policy's performance in the pseudo-real environment throughout the training process. The X-axis represents the num- ber of optimization steps for the policy during training in the simulation, and the Y-axis in- dicates the policy's performance when tested in the pseudo-real environment. In all tasks, our method outperforms UDR and even demon- strates comparable performance to the policy trained directly in the pseudo-real environment.
Cartpole
Go2
200
UDI
0.8
EAS
0.4
125
0.25
0.5
2.0
4.0
Thigh Daf
Ant
Ballbalance
720
600
Ere
830
300
200
400
0.25
2.0
4.0
100
we conduct test to evaluate the performance of EASI across various target vironments. We fix the initial parameter distri- bution ( = U x Er, l x &r] and adjust one of the parameters é to 1/4, 1/2, 2, and times its original value, denoted as G. We choose the proportional parameter of the PD controller for the Cart DoF in Cartpole, the proportional parameter of the PD controller for the Thigh Dof in Go2, the body mass of Ant. and ball mass of Ballbalance as the parameters to be adjusted. We use the simulation with parameter & the target environment and utilize EASI to identify. The performance of trained policies in different target environments are shown in Fig 6 It can be seen that the policies trained with UDR are overly conservative or fail to adapt the target environment due to the gap between source and target environments. On the con- trary, simulators optimized by EASI are more similar to the target environment, thus policies trained
Subsequently,
Body Mass
Bull Mass
Figure 6: Policy performance in target environ- ments with different parameters.
Table 1: Policy performance in target environments with different parameters.
Target Environment
UDR
FineTune
GARAT
EASI
CartPole Cart DoF P x 0,25
116.8
135.147.8
144.4.1.2
164.4.1.66
CartPole Cart DoF
0.5
157.0+10.5
171.218.4
161.614.5
182.912.2
Ant Body Mass x 0.25
588.9-65.9
590.2=18.6
327.7+15.9
780.7=41.5
Ant Body Mass x 0.5
596.4115.3
640.8=30.0
228.4119.2
777.4=37.5

---

in these simulations are more adaptable to target environments and achieve higher performance in the target domain. In specific scenarios, we compare EASI with other sim-to-real methods, with detailed results presented in Table The experimental results demonstrate that policies trained with EASI consistently achieve better performance in the target environment, highlighting EASI's effectiveness in facilitating seamless transfer of policies to the target environment.
5.2 Sim-to-Real Policy Transfer
Cartpole In real-world control, using Cartpole example, state information from robot and sending action commands to the robot. The control frequency for the Cartpole task is 50Hz. We evaluate the performance of the policy by the angle error of pole and the velocity of cart, which reflect the accuracy and stability of the control policy.
the policy PC receiving
Table 2: Policy performance in reality
Algotirhm UDR
Angle Error 10-2
Cart Vel x x10-1
2.65=0.19 1.67=0.16
1.63
EASI
1.21
We train an initial policy using UDR and sample 50 trajectories including approximately 10000 state transitions in real-world environment. For EASI, initial parameter distribution and other algorithm settings remain the same as sim-to-sing tasks. 2 presents the performance in real-world Cartpole tasks, and it can be seen that the policies trained in the simulator optimized by EASI achieve more accurate and stable control in the Cartpole task and show significant improvements compared to previous sim-to-real method.
different algorithms
Go2 To further evaluate the performance of EASI in real-world scenarios, we conduct an experiment using the Unitree Go2 quadruped robot. We search for 7 parameters: the PD parameters for the hip, thigh, and calf joints, as well as the body mass. Unlike the sim-to-sim sim experiment, we do not include ground friction in our search to allow the robot adapt various ground types it might encounter. We train the initial policy using UDR in simulation and use the initial policy to collect real-world data, sampling 2000 steps at control frequency of 50Hz, which provides about 40 seconds of data. Using these real-world samples, we then employ EASI for parameter search.
Galt Frequency
0.01
vel. tracking
0.02
0.01
30
1.0
UDR
EASI
0.00
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
0.2
Thigh
body vl/mmo))
25
0.6
12.5
0.2
sim--EA EAAS
0.2
15
30
35
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
0.4
0.6
0.8
1.0
Proquonoy(Ht))
vdl commands
(a) Movement spectrum of the joint.
(b) Velocity tracking performance.
Figure 7: (a) Using the same policy, there is a significant difference in the motion spectrum of the Go2's joints between the simulation and the real environment due to the reality gap. After optimizing with EASI, the motion spectrum of the Go2's joints in the simulation becomes closer to that in the real environment. (b) Comparison of speed tracking in real environments between policies trained with EASI parameter distribution and trained with UDR parameter distribution.
After parameter search with EASI, the simulator become more realistic, In this experiment, we use the same initial policy and speed command (v=1.4m/s) to control the robot's movement in both

---

simulation and reality environments and plot the frequency spectrum of the robot's joint movements in Fig. 7(a) It is notable that the final output of EASI is the distribution simulator parameters E". To elucidate the differences in simulation with optimized parameter distribution E* and original parameter distribution =(") (consistent with EUDR), we utilize the means of these distributions {* and ( as the parameters for the simulator, referred to as sim-EASI and sim-UDR, respectively. Despite being controlled by the same policy, there are significant differences in the joint movement frequencies between the sim-UDR and real environments, indicating a substantial reality gap between simulator and the real world. After parameter optimization with EASI, the differences in the joint movement frequency spectrum have been significantly reduced, indicating that the sim-EASI more closely resembles real-world.
Subsequently, the real robot's ability to follow speed commands is tested. Using the parameter distribution E' obtained from EASI the simulator parameter distribution to retrain a policy before testing on the real quadruped robot. During the testing phase, we evaluate the robot's performance in the same scenarios using various speed commands. The results in Fig. 7(b) show that policies trained with EASI parameters have better speed tracking capabilities compared to those trained with origin parameters. This also proves that policies trained in the EASIIoptimized simulator can perform better in the real world.
5.3 Real World Data Requirement
We design data budget experiment based on sim-to-sim experiments to test the impact of demon- stration data quantity on EASI. In this experiment, we limit the number of trajectories sampled the target domain to 100, 50, and respectively, and conduct EASI using the limited trajectories.
Table 3: Performance in target environments with varying numbers of reference trajectories
Trajectories
UDR
50
100
200
Cartpole
161.3+15.3
185.1#3.1
182.7+2.7
181.1
182.914.3
Ant Ballbalance
557.44+55.6
724.2+7.5
747.0+225.0
703.1
724.0=31.5
226.9161.77
360.1453.8
421.3426.1
428.5+16.9
393.1 60.5
Go2
0.62+0025
0.44
0.80++0006
0.72+00.5
0.78+0002
The experimental results of the data budget are presented in Table 3 The results demonstrate EASI's data efficiency, as only small amount real-world data is needed for EASI identify parameters that align the simulation closely with the target environment and enhance the policy's 's adaptability in the real world.
6 Limitations
Even though EASI requires relatively low amounts of real-world data, it still necessitates collecting corresponding state transition data for each specific scenario and maintaining certain standards regarding sensor етror ranges. When sensors exhibit significant noise, additional techniques for imitation learning from imperfect demonstrations are needed. Furthermore, even within the same scenario, substantial differences in task styles reduce EASI's 's performance. To achieve optimal results, it is recommended to collect expert data separately for each task.
7 Conclusion
In this work, we introduce EASI, a novel sim-to-real method that combines GAN with ES. EASI uses ES as generator in an adversarial competition with neural network discriminator to find physical parameter distributions that align state transitions between simulation and reality. The discriminator serves as the fitness function for ES, guiding the evolution of parameter distributions. We conducted series of experiments to validate EAST's effectiveness and data efficiency both sim-to-sim and sim-to-real tasks, demonstrating its exceptional performance. We believe EASI is general sim-to-real method characterized by simplicity, low cost, and high fidelity. future work, we will evaluate EASI's performance broader range of robotic platforms and industrial scenarios.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China (Nos. 62073160 & 72394363) and the Nanjing University Integrated Research Platform of the Ministry of Education- Top Talents Program.
References
[1] Xue Bin Peng, Ze Ma, Picter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: adversarial motion priors for stylized physics-based character control. ACM Trans. Graph., 40(4), jul 2021.
[2] Chenhao Li, Marin Vlastelica, Sebastian Blaes, Jonas Frey, Felix Grimminger, and Georg Martius. Learning agile skills via adversarial imitation of rough partial demonstrations. In 6th Anmual Conference on Robot Learning, 2022.
[3] Xue Bin Peng. Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-rea transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3803-3810, 2018
[4] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 23-30, 2017.
[5] Stephen James, Andrew Davison, and Edward Johns. Transferring end-to-end visuomo- tor control from simulation to real world for multi-stage task. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg, editors, Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 334-343, PMLR. 13-15 Nov 2017.
[6] OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in- hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[7] Adam Allevato, Elaine Schaertl Short, Mitch Pryor, and Andrea Thomaz. Tunenet: One-shot residual tuning for system identification and sim-to-real robot task transfer. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 445-455. PMLR, 30 Oct-0l Nov 2020.
[8] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pages 8973 8979, 2019.
[9] Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-to-real transfer. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1290-1296, 2021.
[10] Yifeng Jiang, Tingnan Zhang, Daniel Ho, Yunfei Bai, C. Karen Liu, Sergey Levine, and Jie Tan. Simgan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning. In 2021 IEEE International Conference on Roboties and Automation (ICRA), pages 2884-2890, 2021.
[11] Marius Memmel, Andrew Wagenmaker, Chuning Zhu, Dieter Fox, and Abhishek Gupta. ASID: Active exploration for system identification robotic manipulation. The Twelfth International Conference Learning Representations, 2024.

---

[12] Peide Huang, Xilun Zhang, Ziang Cao, Shiqi Liu, Mengdi Xu, Wenhao Ding, Jonathan Francis, Bingqing Chen, and Ding Zhao. What went wrong? closing the sim-to-real gap via differentiable causal discovery. In Jie Tan, Mare Toussaint, and Kourosh Darvish, editors, Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 734-760. PMLR, 06-09 Nov 2023.
[13] Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain random- ization via probabilistic inference for robotics simulators. arXiv preprint arXiv: 190.00789 2019.
[14] Fabio Muratore, Christian Eilers, Michael Gienger, and Jan Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics and Automation Letters, 6(2):911- 918. 2021.
[15] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, 2019.
[16] Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J. Pal, and Liam Paull. Active domain randomization. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 162--1176. PMLR, 30 Oct-O1 Nov 2020.
[17] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik, Rma: Rapid motor adaptation for legged robots, 2021.
[18] Gwanghyeon Ji, Jubyeok Mun, Hyeongjun Kim, and Jemin Hwangbo. Concurrent training of a control policy and state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, (2))4630-463) 2022.
[19] I Made Aswin Nahrendra, Bycongho Yu, and Hyun Myung. Dreamwaq: Learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning, 2023.
[20] Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, and Vincent Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4243-4250, 2018.
[21] Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcemeen learning. In International Conference on Learning Representations, 2019. 9.
[22] Xingshuo Jing, Kun Qian, Tudor Jianu, and Shan Luo. Unsupervised adversarial domain adaptation for sim-to-real transfer of tactile images. IEEE Transactions on Instrumenaation and Measurement, 72: 1-1,, 2023.
[23] Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized- tewpiiieddt--ccnonnne co-cannnical adaptation networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 12619- 12629, 2019.
[24] Josiah P. Hanna and Peter Stone. Grounded action transformation for robot learning in simula- tion. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. AAAI'1 17, page 3834-3840. AAAI Press, 2017.
[25] Haresh Karnan, Siddharth Desai, Josiah Hanna, Garrett Warnell, and Peter Stone, Reinforced grounded action transformation for sim-to-real transfer, 2020.
12

---

[26] Siddarth Desai, Ishan Durugkar, Haresh Karman, Garrett Warnell, Josiah P. Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2020. Curran Associates Inc.
[27] K.J. Aström and P. Eykhoff. System identification-a survey, Automatica, 7(2):123-162, 1971.
[28] Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification, 2017.
[29] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots, 2018.
[30] Rika Antonova, Jingyun Yang, Priya Sundaresan, Dieter Fox, Fabio Ramos, and Jeannette Bohg. A bayesian treatment of real-to-sim for deformable object manipulation. IEEE Robotics and Automation Letters, 733):5119-5822 2022.
[31] Peng Chang and Taşkin Padif. Sim2real2sim: Bridging the gap between simulation and real- world in flexible object manipulation. In 2020 Fourth IEEE International Conference on Robotic Computing (IRC), pages 56-62, 2020.
[32] Marija Jegorova, Joshua Smith, Michael Mistry, and Timothy Hospedales. Adversarial gen- eration of informative trajectories for dynamics system identification. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7109-7115, 2020.
[33] Allen Z. Ren, Hongkai Dai, Benjamin Burchfiel, and Anirudha Majumdar. Adaptsim: Task- driven simulation adaptation for sim-to-real transfer. In Proceedings of the Conference on Robot Learning (CoRL), 2023.
[34] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[35] Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adver- sarial networks. In International Conference on Learning Representations, 2017.
[36] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, page 2 21--223. JMLR.org, 2017.
[37] Max Schwenzer, Muzaffer Ay, Thomas Bergs, and Dirk Abel. Review on model predictive control: An engineering perspective. The International Journal of Advanced Manufacturing Technology, 11 117(5):1327-1349 2021.
[38] Rakesh P Borase, DK Maghade, SY Sondkar, and SN Pawar. A review of pid control, tuning methods and applications. International Journal of Dynamics and Control, 9:818 8-887,, 18827,20222 2021.
[39] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.
[40] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018
[41] Huiqiao Fu, Kaiqiang Tang, Yuanyang Lu, Yiming Qi, Guizhou Deng, Flood Sung, and Chunlin Chen. Ess-infoGAIL: Semi-supervised imitation learning from imbalanced demonstrations, In shirth-eeventh Conference on Neural Information Processing Systems, 2023.
[42] Nikolaus Hansen. The ema evolution strategy: tutorial, 2023.
[43] Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Natural evolution strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pages 3381-3387, 2008
[44] Hans-Georg Beyer. The Theory of Evolution Strategies. Springer Berlin Heidelberg, 2001.
13

---

Supplementary
A Full parameter search
To explore the performance of EASI as the number of simulation parameters increases, we conducted sim-to-sim experiment with the Go2 environment, searching for 25 parameters. In the experiment, we searched for the PD parameters of all motors in the Go2 robot and the mass of the robot. The results of the parameter search are shown in Fig. g (Due to space limitations, 10 out of 25 results displayed).
Go2 Param Search
50
92
50
20
WD
200
92
20 40
40
20 Gererudon
20
42
Clnuur soin
Figure 8: The convergence process for different parameters being searched in the Go2 environment. WD means target parameter within initial parameter distribution, while OOD means out of initial parameter distribution.
B Hyperparameter sensitivity analysis
Since EASI uses Evolutionary Strategies (ES) as the simulation parameter generator, it is much more stable in the training process compared to the original GAN algorithm and less sensitive to hyperparameters. In EASI, we select two hyperparameters' for sensitivity analysis in Table and Table S the evaluation metric the L2 error between the searched parameter and the target parameters.
We first analyze the hyperparameter p/ In each evolution, we test À individuals and select the best-performing μ elite individuals for recombination and mutation. Generally, higher // À value helps maintain diversity in the population but may slow down the evolution process. Conversely, if n/> is too low, lack of diversity can lead to local optima. To enhance parameter search, recommend using larger n// and increasing the number of evolution steps, which will improve EASI's performance but also increase computational time.
Next, we analyze another hyperparameter epoch_dii disc, which refers to the number of times the discriminator is trained before each generation of evolution. Insufficient training may prevent the discriminator from achieving optimal performance, while excessive training can cause overfitting. In experiments, found that varying epoch disc had only minor effect the final search results.
Table 4: Hyperparameter sensitivity analysis for EASI (H/A).
p/A
0.1
0.3
0.5
0.7
0.9
Param Search Error
1.35
0.63
0.34
1.99
1.44
0.04
Table 5: Hyperparameter sensitivity analysis for EASI (epoch__ise)
epoch disc
10
15
20
Param Search Error 0.60 0.008
0.19+F0.14
0.34
0.20
0.82
0.46=00.04
14

---

C
Broader Impacts
In our work, have proposed EASI for simulation trained strategy transfer reality. This could potentially drive the application of robotics technology in the real world, thereby positively impact- ing societal operational efficiency. Sim-to-reat techniques reduce the need for extensive physical trials, minimizing costs associated with hardware testing and prototyping. This can accelerate the development of robotic technologies, making them more accessible to a broader range industries.
However, there is a risk that developers may become overly reliant on simulation results, potentially leading systems that perform well in simulated environments but fail in the real world due to unmodeled complexities. Additionally, it is important to recognize that the capabilities of simulations have limits, and we cannot expect simulations perfectly mirror reality. Resources should still be invested in designing better robot structures and developing more advanced simulators.
Moreover, there growing concerns about the computational resources required for current training schemes. The increasing demand for computational power undoubtedly consumes significant amounts of energy and resources. We are continually exploring more efficient algorithms to address this challenge.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the main contributions, including the key method and scope of the research, which supported by the theoretical and experimental results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes dedicated "Limitations" section where we discuss the potential weaknesses of our approach.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
Justification:
This paper primarily focuses on the innovative combination of existing methods
and their application in sim-to-real scenarios, and does not involve proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The method proposed in this paper is easy to reproduce, and we introduce detailed information of method implementation and experiment settings in Section 5,
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We in the process of organizing the experiment code and we will provide open access to well-documented code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic//uuides/Coeeeubbiis casssiionoolyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttp:/cccppplcguuuaieeaeaecc Stteeee//ooSSSeeissSioica www.ssssiicgcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant training and test details are described in Sections 5.1, 5.3, and supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are reported for key results. All figures and tables include erтor bars capturing the variability due different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies the type of compute resources used, including CPU and GPU specifications, etc. These details are provided in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httttps://eeurrp..cpppbbbbiclee insee
Answer: [Yes]
Justification: The research adheres the NeurlPS Code of Ethics, with careful consideration given to ethical standards throughout the study.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: The potential socictal impacts, both positive and negative, are discussed the Appendix C
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The method proposed in this paper is mainly applied to robot control tasks and poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in this research are properly credited, and their licenses and terms of use are respected. This information is provided in the references and acknowledgment sections.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thoode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets, including datasets and code, are thoroughly documented, and this documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "6wIUAruJZe", "reviewer": "Reviewer_rASM", "summary": "The paper introduces Evolutionary Adversarial Simulator Identification (EASI), a novel sim-to-real method using a combination of Generative Adversarial Network (GAN) and Evolutionary Strategy to bridge the gap between simulation and reality. EASI optimizes simulator parameters by aligning state transitions between simulation and reality, enhancing policy transferability. The method is tested on various tasks, demonstrating superior performance and stability compared to existing sim-to-real algorithms through experiments.", "questions": "1) How do you capture/model the state transition, since the sim-transitions will be used to measure the gap with the real-transitions by JS?\n2) In algorithm lines 9-10, Do they require the same initial state s_k? If yes, how does the work make sure they are aligned? If not,  how do the authors measure the transition divergence? I assume the JS divergence was employed (line 157), it would require two distributions, how do the authors obtain/estimate the real-transitions distribution? \n3) The baseline methods are only the Oracle and Uniform Domain Randomization(UDR), can authors incorporate the GAT branch methods, and Domain Adaptation methods as well? \n4) I have concerns about the generalizability of the proposed method since for GAN-based methods, it is notoriously difficult to train, especially for high-dimensional feature space, and what is presented in the paper seems still a relatively simple environment. \n5) Can authors explain If Fig.3 reflects the training of proposed methods in real env or simulator env? If it is trained in a real env, it doesn’t make sense for sim-to-real transfer.", "limitations": "1) The paper lacks proof of the high-dimensional transfer performance either empirically or theoretically. \n\n\n2) And the work seems not able to easily adapt to new tasks, it requires training all over again (using the GAN paradigm) in a new real-world environment.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The paper presents a novel method to mitigate the sim-to-real performance gap for Reinforcement Learning algorithms, which has an influential impact on the practical meaning. \n\n\n(2) It adopts the Generative Adversarial Learning paradigm, designs a fitness function as a discriminator to identify the parameter distribution, and trains an Evolution-based algorithm as a generator to search for the system parameters.", "weaknesses": "1) This paper lacks explanations on important technique details. Such as how the transition distribution is derived from the demonstration data. \n2) Some parts of the paper's demonstration are confusing, for example, if Fig.3 reflects the training of proposed methods in real env or simulator env. If it is trained in a real env, it doesn’t make sense for sim-to-real transfer.\n3) The paper discussed comprehensive related work, but very few baseline methods were compared regarding the performance in the paper, such as Grounded Action Transformation, etc. \n4) The setting of the experiment is not as convincing as what is claimed when targeting the existing work’s problems. Such as line 42-43, other methods are hard to perform well in large-dimensional parameter space, this paper only conducted experiments on the environment with a maximum of 11 parameters, which might still be simple env. \n5) The paper claimed `yes` in checklist 5, open-access of code, but it is not available anywhere in the paper, which is not aligned with the checking claims, and may hinder the reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer rASM\n`Q1. How do you capture/model the state transition`\n\n`A1.` In this paper, we draw on the training paradigm of GANs to estimate the distance between the simulated and real state transition distributions. According to GAN theory [1], Discriminator D and Generator G play the following two-player minimax game with value function $V(G,D)$:\n\n$$\\mathop{min}\\limits_{G} \\mathop{max}\\limits_{D}V(G,D)=E_{x\\sim p_{data}(x)}[logD(x)]+E_{z\\sim p_{z}(z)}[log(1-D(G(z)))].$$\n\nFor any given fixed G, training D to maximize the\nquantity $V(G,D)$, we can proof that [1]:\n\n$$C(G)=\\mathop{max}\\limits_{D}V(G,D) =-log(4)+KL(p_{data}||\\frac{p_{data}+p_{g}}{2})+KL(p_{g}||\\frac{p_{data}+p_{g}}{2})$$\n$$C(G)=-log(4)+2\\cdot JSD(p_{data}||p_{g}).$$\n\n$C(G)$ could view as the estimication of the Jensen–Shannon divergence between two distributions. \n\nWe could leverage $C(G)$ to estimate the distance between state transition distributions. In each generation of evolution, **we only need to randomly sample $(s,a,s')$ tuples** from real state transition and simulated state transition to train the discriminator. Once the discriminator reaches its optimal, it can be used to estimate the distance between real and simulated state transition distributions.\n\nIn EASI, we use a WGAN-style discriminator\n\n$$C(G) = \\mathop{\\max}\\limits_{D} E_{d^{\\mathcal{M}}(\\mathbf{s},\\mathbf{a},\\mathbf{s}')}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')]-E_{d^{\\mathcal{B}}{(\\mathbf{s},\\mathbf{a},\\mathbf{s}')}}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')],$$\n\nto tackle the discriminator vanishing gradients problem, and for this instance, $C(G)$ approximate the Earth-Mover distance between two distributions [2]. \n\n[1] Goodfellow, Ian, et al. \"Generative adversarial nets.\" Advances in neural information processing systems 27 (2014).\n[2] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. \"Wasserstein generative adversarial networks.\" International conference on machine learning. PMLR, 2017.\n\n\n`Q2. Do they require the same initial state s_k?`\n\n`A2.` **EASI does not have specific requirements for the initial agent state $s_k$, nor does it require the alignment of trajectories and state transitions.** In fact, this is a key advantage of EASI. EASI uses discriminator to estimate the distance between the state transition distributions of simulation and demonstration. Unlike previous work, which required aligning sim and real trajectories and then comparing trajectory errors, EASI uses $(s,a,s')$ tuples to train a discriminator and estimate the distance of state transition distribution by the discriminator.\n\n`Q3. Can authors incorporate the GAT branch methods, and Domain Adaptation methods as well?`\n\n`A3.` In the **common response A4**, we introduced additional comparative experiments, including the FineTune and GARAT (a GAT branch method). In these experiments, EASI consistently demonstrated the best performance in transferring simulation trained policies to the target environment.\n\n`Q4. The generalizability of the proposed method`\n\n`A4.` Since EASI uses Evolutionary Strategies (ES) as the simulation parameter generator, it is much more stable in the training process compared to the original GAN algorithm and less sensitive to hyperparameters. In **common response A5**, we conducted a hyperparameter sensitivity analysis for EASI. The results showed that EASI has low sensitivity to hyperparameters, further demonstrating its stability. \n\nAdditionally, We tested EASI's sim-to-real performance in the real world using the Unitree Go2 robot in **common response A2**. The task in this experiment is complex and challenging, and EASI is still able to exhibit the expected performance.\n\n`Q5. Can authors explain Fig.3`\n\n`A5.` The training was conducted entirely in the original sim environment, and we saved models from various stages of the training process. These saved models were then tested in the target environment (since it's sim-to-sim, model testing was very convenient). The x-axis represents the number of policy updates in the original sim environment during RL training, while the y-axis shows the performance of the saved model tested in the target environment.\n\nThrough this figure, we want to show that EASI not only facilitates better transfer of policies trained in the original environment to the target environment, but also accelerating the training process because of the narrower Domain Randomization (DR) parameters range. We apologize for the confusion caused by Fig. 3. In future versions, we will include additional explanations to ensure Fig. 3 is better understood.\n\n`Q6. Code is not available anywhere in the paper.`\n\n`A6.`  We have uploaded all the [codes](https://anonymous.4open.science/r/EASI-0998/) and supplementary materials.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b4QJKbBS8m", "reviewer": "Reviewer_2SzS", "summary": "The paper tackles the issue of finding correct parametrizations for simulators for robot tasks in order to close the sim-to-real gap. To optimize the simulation parameters, evolutionary strategies (ES) are employed in combination with a discriminator function (trained in a GAN setting). The study shows that the presented approach yields better results (running in whatever setting is considered \"real\") than the baseline approach Uniform Domain Randomization (UDR).", "questions": "Eq. 4 is unclear to me. Does it contain a typo?", "limitations": "The limitation section is very good to have, but quite weak as limitation sections go.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper tackles the important topic of closing sim-to-real gaps, which might have impat eve way beyond robots. The employed method appears rather straightforward, which is a big plus, since it means that either (i) it is in fact the natural method to use or (ii) the authors did such a good job sharing their perspective that the reader almost thought of the siolution him/herself. In either case, the paper shows empirical success by outperforming UDR.\n\nThe choice of domains seems appropriate and the study shed a light on the various behavorial properties of the presented approach. Albeit limited in space, the authors try to show limitations and problems of the approach as well. It is very important that the authors not only show the result but (with studies like that one behind Fig. 2, e.g.) they also provide some analysis on why things work.\n\nI enjoyed that at multiple points throughout the paper, the current agenda is repeated and it is always clear what the authors want to show.", "weaknesses": "Some connections within the paper are very confusing. Alg. 1 is not properly referenced und thus not discussed deeply enough. Fig. 1 is not referneced properly and comes way too late within the paper. Various explanations on GANs are confusing as -- in this approach -- we do not have a generator in its own right but build a slightly different architecture. The plots in Fig. 2, e.g., are explained at various non-continuous locations and thus very hard to follow.\n\nThe study lacks some stronger comparison, but if the field does not have much to offer here at the moment, this is acceptable. I am missing some disucssion about the comparative use of compute. EASI seams much more involved than UDR here.\n\nThe limitation section is very good to have, but quite weak as limitation sections go.\n\nEq. 4 is unclear to me. Does it contain a typo?\n\nSeveral formal issues persist, including the usage of abbreviated forms (\"it's\"), various references without a qualifier (\"according to 4\"), and missing spaces. I recommend thorough proofreading. I also find it to be more common to use \"an RL problem\" and \"an MDP\".\n\nSome more practical illustrations on the impact of the method would have been nice, especially the real world experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 2SzS\n`Q1. The study lacks some stronger comparison. `\n\n`A1.` In the **common response A4**, we introduce additional comparative experiments, including the FineTune and GARAT. In these experiments, EASI consistently demonstrates the best performance in transferring simulation trained policies to the target environment.\n\n`Q2. The comparative use of compute.`\n\n`A2.` On our platform, we use EASI for one of the most complex tasks, Go2. EASI completed the parameter search in less than 10 minutes and is able to adjust the simulation parameter distribution to an appropriate range. After determining the simulation parameter distribution, UDR is used for policy training or fine-tuning.\n\nWhile using UDR directly can save the time spent on parameter search, as mentioned in the paper, we often lack the knowledge to set a reasonable parameter range for UDR. Without specialized knowledge to adjust the UDR parameter range, the trained policy might not transfer well to the real environment. Thus, directly using UDR may require extensive expertise and trial and error, which can be time-consuming compared to using EASI. By using EASI, we can leverage a small amount of real-world demonstration to quickly determine the UDR simulation parameter range, avoiding trial and error and improving the performance of the sim-to-real policy.\n\n`Q3. Some connections within the paper are very confusing. `\n\n`A3.` We carefully reviewed the issues you've raised, and we consider your feedback both valuable and important. We will promptly adjust the layout of the images in the paper and provide additional explanations for Fig. 1, offering a more detailed introduction to its contents. For Fig. 2, we will modify the layout of the paper to better link the legend to the corresponding content. Regarding the unclear phrasing, we will also make corrections in the next version, ensuring that readers can more easily understand our intended message. Thank you for your insightful feedback!\n\n`Q4. Eq. 4 is unclear to me. Does it contain a typo?`\n\n`A4.` The objective of Eq. 4 is to find a discriminator D that can better distinguish between simulated data and real data. We have revised Eq. 4 to make it more rigorous:\n\n$$\\mathop{\\max}\\limits_{D} [E_{d^{\\mathcal{M}}(\\mathbf{s},\\mathbf{a},\\mathbf{s}')}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')]-E_{d^{\\mathcal{B}}{(\\mathbf{s},\\mathbf{a},\\mathbf{s}')}}[D(\\mathbf{s},\\mathbf{a},\\mathbf{s}')]].$$\n\nThank you for raising these issues. We will promptly revise the paper and add additional explanations to make the formula expressions clearer and easier to understand.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iN2aDjSrWy", "reviewer": "Reviewer_zVKJ", "summary": "This submission presents an approach to sim2real transfer by predicting better simulation hyperparameters via Evolutionary Adversarial Simulator Identification (EASI). EASI is a combination of a generative adversarial network (GAN) and evolutionary strategy approach to generating simulation hyperparameters. Concretely however in the GAN setup, the generator is not another neural network, only the discriminator is a neural network that predicts whether a state transition $(s, a, s')$ is real or not. The generator is now an evolutionary search procedure that generates a distribution of simulation hyperparameters that strives to maximize discriminator rewards. Importantly EASI does not simply generate fixed simulation hyperparameters, but rather a range/set of hyperparameters and can be seen as a more \"educated\" domain randomization approach when used for training. The result of EASI is better sim2real transfer of policy performance when compared to a standard uniform domain randomization technique. Empirical tests are on robotics control tasks tested can match the oracle.", "questions": "Questions:\n- How come in ball balance the oracle method (SAC training on dense rewards with the correct sim parameters) performs worse than EASI, which has SAC training on sim parameters predicted by rough UDR.\n- Are dense rewards used for all tasks?\n- What if UDR does not get any successful demonstrations e.g. in a cartpole task say the pole is never swung up completely. It sounds like in this scenario EASI might not generate a good set of hyperparameters that would be important for unseen parts of the task?\nTypos:\n- What does \"domain priority\" in section mean? The line \"However, DR needs specific domain priority and hand-engineer to determine the\n79 distribution of injected random noise.\" might have typos and grammar issues", "limitations": "A limitation section is provided and adequately points out the assumption that EASI relies on expert trajectories and accurate state estimation of the same observation data in simulation and the real world.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- A very unique approach to sim2real by finding ways to generate a range/set of hyperparameters to try and test on. The approach is very lightweight and is built on top of an initial uniform domain randomization (UDR) + training procedure to then find a better set of simulation hyperparameters.\n- Equations and losses are clearly laid out and well-motivated. The algorithm is easy to understand and reason with.\n- The performance of EASI looks great. I would hypothesize that UDR fails to do as well as EASI for some tasks because the RL policy spends too much time training on environments with hyperparameters too far away from the real world / don't help with sim2real transfer. It would be good to see additional analysis on why exactly UDR struggles a lot compared to EASI in terms of performance under a fixed online budget.", "weaknesses": "- Only control tasks are tested which generally are easier to solve with just domain randomization (albeit EASI does better here). It would be useful to see more complex tasks beyond simple control tasks that leverage manipulation as well. This point doesn't lower my score but if more complex tasks were analyzed I'd be happy to raise the score further to reflect the bigger impact of the method.\n- To leverage EASI some sim+real data is required still to train the discriminator and run the evolutionary search. While the amount of real data in experiments can be kept to just one expert demonstration, it's not clear how much simulated data is needed, and what distribution of simulated data is needed. Section 5 mentions UDR is used to train a rough policy but how rough is rough? What if UDR does not get any useful data / what would be considered sufficient UDR to then kickstart EASI?\n- Since EASI has to run UDR itself first to get some sim data to train on, overall it could be that running UDR for as long as it takes to run UDR + EASI achieves the same performance, although Figure 3 suggests otherwise. There don't seem to be figures/information about how long the first UDR is done for EASI. It might be fairer to run the UDR baseline for as many online steps EASI uses for training after EASI and before with the rough UDR.\n- As pointed out in the limitations section, this sim2real setup requires measuring state in the real world and simulation. For control tasks where the state often is just the robot state and can be measured via sensors, this is fine. For more complex tasks involving manipulation of other objects, this approach will probably not work well.\n\nHappy to raise my score if the above and questions are addressed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer zVKJ\n\n`Q1. How come in ball balance the oracle method performs worse than EASI.`\n\n`A1.` BallBalance is a unique environment where the ball can only be controlled indirectly by a movable table and cannot be directly influenced by actions. In this task, if the ball falls off the table, the episode ends and a penalty is incurred. We hypothesize that during the early stages of ORACLE training, the policy may prioritize maximizing rewards by keeping the ball at the target position, which could inadvertently lead to the ball falling. In contrast, EASI narrows the parameter range but still allows for variations across different environments. As a result, the policy may adopt a more cautious approach in the early training stages, focusing on preventing the ball from falling off and thus avoiding penalties. As training progresses, both ORACLE and EASI can effectively keep the ball from falling while maintaining its position at the target, leading to similar performance in the later stages of training.\n\n`Q2. Are dense rewards used for all tasks?`\n\n`A2.` Yes, all experiments were conducted using dense rewards during training. EASI mainly focuses on the state transitions of these tasks, and the training method of the task itself is not the primary focus of EASI.\n\n`Q3. What if UDR does not get any successful demonstrations? `\n\n`A3.` For EASI, our requirement for realworld demonstration is that the $(s,a,s')$ tuples contain sufficient information, meaning that the state transition distribution in the demonstration needs to partly overlap with that in the simulation. If the rough UDR policy can run in the real environment, even if the performance is not ideal, it can still provide valuable state transitions for EASI training.\n\nIf UDR completely fails to accomplish the task, but some meaningful $(s,a,s')$ tuples are still recorded during action execution. In this situation, the collected demonstration data may be imperfect and unbalanced, with only a small portion of state transition overlap with that in the simulation. In such cases, EASI may fail due to insufficient state transition information.\nWe mentioned this limitation in the paper, and notice that current imitation learning methods address imperfect and unbalanced demonstrations. In the future, we will aim to use imperfect and unbalanced imitation learning methods to extract useful state transition information from failed real-world demonstrations and then use EASI to find suitable parameters. We believe that in future work, EASI could have lower requirements for real demonstrations, allowing it to identify optimal parameters even when the policy complete very poorly in the reality.\n\n\n`Q4. What does \"domain priority\" in section mean? `\n\n`A4.` What we meant to express is that Domain Randomization (DR) requires prior knowledge of the specific domain, including detailed parameters about real-world deployment. Thank you for pointing out the issues with the wording in the paper, and we will revise the wording to ensure clearer understanding.\n\n\n\n`Q5. Only control tasks are tested which generally are easier to solve with just domain randomization.`\n\n`A5.` The gap between simulation and real-world environments has long been a significant challenge for deploying RL-based robotic controllers in practical applications. Through our method EASI, we aim to advance the deployment of such controllers, focusing specifically on control tasks. And additionally, we tested EASI in a more complex sim-to-real tasks using the Unitree Go2 robot in **common response A2**.\n\n`Q6. it's not clear how much simulated data is needed.`\n\n`A6.` During EASI's process, $\\lambda$ environments with different parameters are created in the Isaac Gym simulator (in the experiment, we set $\\lambda=300$), and use a same policy to collect 1 trajectory for each sim environment, the max_trajectory length is setted in different tasks. Thanks to Isaac Gym's high degree of parallelism, we can parallelly run hundreds or even thousands of environments with different physical parameters. After each evolutionary iteration, all environments are reconfigured with new physical parameters based on the evolutionary strategy, initiating the next evolution cycle. The process of generating sim data is highly convenient, fast, and inexpensive, so the amount of simulation data used is not a major concern.\n\n`Q7. It might be fairer to run the UDR baseline for as many online steps EASI uses for training after EASI and before with the rough UDR.`\n\n`A7.` To ensure fairness, we used the same real demonstration data as EASI to fine-tune the UDR algorithm. The specific experimental details are described in **common response A4**. In more complex environments like Ant, FineTune's performance improvement is limited when the amount of real demonstration data is insufficient. EASI, on the other hand, consistently achieves significant improvements in policy transfer to target environment.\n\n`Q8. For more complex tasks involving manipulation of other objects, this approach will probably not work well.`\n\n`A8.` For complex tasks involving manipulated objects, if the parameters of the object are known, our ball balance experiment can be seen as such a task, where a 'manipulated ball' is placed on the table. If the parameters of the manipulated object are uncertain, we can still utilize EASI to adjust the manipulator's parameters, allowing the simulation to more closely resemble the target environment while accommodating a wider range of parameters for the manipulated object.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sZQyJs9yKn", "reviewer": "Reviewer_ZLtE", "summary": "This work studies the sim-to-real transfer using an evolution strategies with a learned discriminator. The discriminator learns to distinguish the state transition between real and simulation. The evolution strategies aim to optimize the parameters of the simulator so that the data generated by the simulator is undistinguishable by the discriminator. Experiments on sim-to-sim and sim-to-real cases have been conducted.", "questions": "1) It seems that the major technical contribution of this work is to learn an objective function for the ES using GAN, for simulator calibration or say parameter inference. Thus, I am very curious about the architecture of the used GAN and how are the tuple s, a, s' processed in the training. More descriptions on the details of used GAN are welcome.\n2) In Figure 3, why the EASI can outperform the oracle that indicates the upperbound of the performance?\n3) For sim-to-transfer case, the oracle may not represent the upperbound, in which case some advanced compared algorithms need to be compared in the experiments.\n4) It is interesting to see EASI can adjust the parameters out of the initial range. What makes it capable of doing so?\n5) How is the scalability of the EASI, I mean if the number of parameters increases, what will happen to EASI?", "limitations": "YES", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed EASI framework is easy to understand and reasonable. The three research questions listed in the experiments are important and have been demonstrated.", "weaknesses": "The details of the employed GAN architecture lacks furture description.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer ZLtE\n`Q1. The architecture of the used GAN and how are the tuple s, a, s' processed in the training. `: \n\n`A1.` The input to the discriminator consists of a state transition $(s, a, s')$. This $(s, a, s')$ data is combined into a tensor with a shape of $(2 \\times \\text{dim(state)} + \\text{dim(action)})$, which is then fed into a multilayer perceptron (MLP) discriminator. The discriminator is trained using a WGAN-based approach to determine whether the $(s, a, s')$ data originates from the demonstration. Subsequently, we employ Evolution Strategies (ES) to select parameters that make the simulation closely resemble the real environment and to 'generate' the next generation of environment parameters. Unlike conventional GANs, EASI does not utilize a neural network generator; instead, it relies on ES to choose and generate the next set of parameters based on the discriminator's output. Thanks to the stability of ES, EASI achieves faster and more stable training compared to traditional GANs.\n\nIn our experiments, an MLP discriminator is sufficient to achieve good results. We also considered employing networks such as LSTM or TCN for processing $(s, a, s')$ data. However, since the MLP has already demonstrated exceptional performance in the current tasks, we plan to explore these alternative network architectures in future work.\n\n`Q2. In Figure 3, why the EASI can outperform the oracle that indicates the upperbound of the performance?`\n\n`A2.` BallBalance is a unique environment where the ball can only be controlled indirectly by a movable table and cannot be directly influenced by actions. In this task, if the ball falls off the table, the episode ends and a penalty is incurred. We hypothesize that during the early stages of ORACLE training, the policy may prioritize maximizing rewards by keeping the ball at the target position, which could inadvertently lead to the ball falling. In contrast, EASI narrows the parameter range but still allows for variations across different environments. As a result, the policy may adopt a more cautious approach in the early training stages, focusing on preventing the ball from falling off and thus avoiding penalties. As training progresses, both ORACLE and EASI can effectively keep the ball from falling while maintaining its position at the target, leading to similar performance in the later stages of training. \n\nOn the other hand, UDR, with its overly wide parameter range, leads to an overly conservative policy. This policy only achieves the basic reward of preventing the ball from falling off but is hard to maintain the ball at the target position.\n\n`Q3. For sim-to-transfer case, the oracle may not represent the upperbound, in which case some advanced compared algorithms need to be compared in the experiments. `\n\n`A3.`In the **common response A4**, we introduced additional comparative experiments, including the FineTune and GARAT (a GAT branch method). In these experiments, EASI consistently demonstrated the best performance in transferring simulation trained policies to the target environment.\n\n`Q4. It is interesting to see EASI can adjust the parameters out of the initial range. What makes it capable of doing so?`\n\n`A4.`In EASI, we utilize Evolution Strategies (ES) to adjust the distribution of parameters. During the evolution process, ES incorporates mechanisms such as \"recombination\" and \"mutation.\"\n\nFor recombination: Parameters that contribute to a more realistic environment have a higher probability of producing offspring. As a result, parameters within the initial range that are closer to the target value exert greater influence on the next generation's parameter range, helping to refine the distribution towards the target. For mutation: Each time new offspring are generated, some undergo mutations that deviate from the original distribution. If these mutated individuals achieve higher fitness values than those within the original distribution, the population will gradually evolve in the direction of the mutated individuals. This means that even if the initial parameter range does not include the global optimum, EASI can potentially discover new parameters close to the optimum through multiple iterations.\n\nAfter refining the simulator's parameter distribution with EASI, we can train or fine-tune the policy in this enhanced simulator, thereby improving sim-to-real performance.\n\n`Q5. How is the scalability of the EASI, I mean if the number of parameters increases, what will happen to EASI?`\n\n`A5.` In **common response A3**, we introduced additional simulation parameters, searching for a total of 25. EASI was still able to identify the appropriate target parameter range.\n\nIn our real-world Go2 sim-to-real experiment discussed in **common response A2**, we only searched across 7 parameters. We set the motors on the four legs to the same positions, resulting in identical parameters for the hip, thigh, and calf motors, each with two PD parameters. Despite this limited search, we achieved excellent sim-to-real performance. By focusing on a few key parameters with EASI, we were able to significantly minimize the gap between the simulation and the real environment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QAPXM9q8qU", "reviewer": "Reviewer_yoqm", "summary": "The paper introduces an approach to address the challenges of transferring reinforcement learning (RL) policies from simulation to real-world applications. Traditional methods like Domain Randomization (DR) require significant domain-specific knowledge and extensive training time, which often result in suboptimal performance. EASI combines Generative Adversarial Networks (GAN) and Evolutionary Strategies (ES) to identify physical parameter distributions that align state transitions between simulation and reality, thus minimizing the reality gap.", "questions": "1. **Parameter Sensitivity and Selection**:  \n\n   - **Question**: How sensitive is EASI to the choice of hyperparameters for both the GAN and the ES components?  \n\n   - **Suggestion**: Provide a detailed sensitivity analysis or guidelines for selecting these parameters to help with replicability and optimization in different settings. \n\n   \n\n2. **Diversity of Experimental Tasks**: \n\n   - **Question**: How does EASI perform across more complex real-world scenarios? \n\n   - **Suggestion**: Expand the experimental validation to include other real-world applications (for example, navigation) to strengthen claims about the method’s general applicability. \n\n   \n\n3. **Computational Resources**: \n\n   - **Question**: What are the computational requirements for training GANs and performing evolutionary searches in EASI? \n\n   - **Suggestion**: Discuss the computational costs and potential optimizations to make the method more accessible for practical deployment. \n\n  \n\n4. **Scalability**: \n\n   - **Question**: How scalable is EASI to high-dimensional state and action spaces? \n\n   - **Suggestion**: Address the scalability of the method and include preliminary discussions on scaling to more complex systems.", "limitations": "The limitations are thoroughly discussed in Section 6.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "#### Originality \n\n- **Approach**: Combining Generative Adversarial Networks (GAN) with Evolutionary Strategies (ES) to address sim-to-real transfer is creative. \n\n#### Quality \n\n- **Low Data Requirement**: The method's effectiveness with minimal real-world data is a significant advancement. \n\n#### Clarity \n\n- **Clear Exposition**: The paper is well-written and explains complex concepts clearly, making it accessible to a broad audience.  \n\n#### Significance \n\n- **Practical Impact**: EASI addresses the reality gap in robotics and reinforcement learning, providing a cost-effective and efficient solution with significant practical implications. \n\n- **Broad Applicability**: The method's success in various tasks suggests potential for wide adoption across different domains.  \n\nIn summary, the paper offers an interesting solution to a critical problem in reinforcement learning.", "weaknesses": "#### Methodological Concerns \n\n1. **Wasserstein GAN loss**: The Wasserstein GAN loss was first introduced by the Wasserstein GAN. Using the Wasserstein GAN loss with clipped network weights in the range [-0.01, 0.01] does not seem to be a novel contribution by the author. \n\n  \n\n#### Experimental Limitations \n\n1. **Diversity of Tasks**: While the paper presents experiments on various tasks, the real-world applications are somewhat limited. For instance, in Sim to Real experiments, only Cartpole is used. Expanding the experiments to include more complex real-world scenarios, such as robot dogs or navigation tasks, would strengthen the claims about EASI's broad applicability. \n\n  \n\n#### Practical Considerations \n\n1. **Computational Resources**: The computational cost of EASI, especially for training GANs and performing evolutionary searches, is not thoroughly discussed. Providing insights into the computational requirements and potential optimizations would be valuable for practical deployment. \n\n2. **Scalability**: The scalability of EASI to high-dimensional state spaces and action spaces is not well-addressed. Including discussions or preliminary results on scaling the method to more complex systems would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer yoqm\n\n`Q1. How sensitive is EASI to the choice of hyperparameters for both the GAN and the ES components?`\n\n`A1.` We conducted a hyperparameter sensitivity analysis for EASI, as detailed in **common response A5**, and provided recommendations for parameter settings. EASI has low sensitivity to hyperparameters, and we rarely changed EASI's hyperparameters across all experiments in this paper. We believe that EASI can achieve plug-and-play effectiveness in other tasks.\n\n`Q2. How does EASI perform across more complex real-world scenarios?`\n\n`A2.` We tested EASI's sim-to-real performance in the real world using the Unitree Go2 robot in **common response A2**. The Go2 task is complex and challenging, yet EASI is able to adjust the simulation parameters using approximately 40 seconds of real demonstration data, making the simulation more realistic.\n\n`Q3. What are the computational requirements for training GANs and performing evolutionary searches in EASI?`\n\n`A3.` EASI's computation consists of three main parts: sim trajectory collection, discriminator training, and parameter evolution.\n1. Sim Trajectory Collection: This is the most computational-consuming step. Assuming we have $\\lambda$ offspring in the evolutionary strategy, we need to use same policy to collect $\\lambda$ trajectories from environments with $\\lambda$ different parameters. Due to the high parallelism of the Isaacgym simulator, we can generate $\\lambda$ environments with different parameters in Isaacgym and sample $\\lambda$ trajectories concurrently. \n2. Discriminator Training: This involves sampling from both demonstration and environment trajectories to train the discriminator.\n3. Parameter Evolution: In this part, we use the discriminator to estimate the similarity between the simulation trajectory and the demonstration. Then, using ES to generate the distribution of the next generation of simulation parameters. Based on this new parameter distribution, we sample and obtain the physical parameters of N simulated environments and set the parameters for each environment.\n\nIn our experiments with the most complex environment, Go2, setting $\\lambda$=300 and trajectory_length=500，we test the time consuming on a PC equipped with Intel i5-13600KF and RTX4060 Ti. For one generation of evolution, the time consuming is as follows:\n\n|  process   | time consuming(s)  |\n|  :------: | :------:  |\n| sim trajectory collection  | 6 |\n| discriminator training  | 0.09 |\n| parameter evolution  | 0.11 |\n\nTypically, 30 to 50 generations are sufficient for parameter convergence. \n\n`Q4. Discuss potential optimizations to make the method more accessible for practical deployment.`\n\n`A4.` Currently, EASI has demonstrated strong performance for real-world deployment. However, we believe there is still significant potential for improvement. Real-world data collection often lacks the ideal conditions found in simulations, and the rough policy trained with UDR may not perform optimally in practical applications. As a result, the demonstration data we gather in real-world settings may be imperfect and unbalanced. As noted in the limitations, there are existing imitation learning methods designed to handle unbalanced and imperfect demonstrations. In the future, we aim to implement similar approaches to train a more effective discriminator based on these imperfect, unbalanced datasets, thereby reducing the deployment costs associated with EASI.\n\n`Q5. How scalable is EASI to high-dimensional state and action spaces?`\n\n`A5.` In the Go2 sim-to-real experiment detailed in **common response A2**, the task featured high-dimensional state and action spaces. Despite these complexities, EASI demonstrated excellent performance in this sim-to-real task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Common Response
We sincerely appreciate the reviewers for their insightful feedback! In the following, we will first address the comments that are shared by multiple reviewers.

 `Q1. The significance of this work.`

 `A1.` Domain randomization (DR) has become one of the most popular sim-to-real algorithms due to its simplicity and adaptability for various robotic applications. EASI introduces a new method for sim-to-real that quickly identifies the parameter range for DR, **creating a more realistic simulation environment for policy training**. With its efficiency and ease of use, we believe EASI can serve as a valuable tool for  the sim-to-real processes.

`Q2. More complex real-world applications.`

`A2.` To test the performance of EASI in the real world, we conduct an experiment with the Unitree Go2 quadruped robot. The goal is to have the robot move in a canter gait based on speed commands from a remote controller. The inputs for this task include 57-dim proprioceptive observations, 4-dim estimated privileged observations, 29-dim historical encodings, 570-dim historical observations, and 7-dim control commands. The outputs are the target positions for the robot's joint angles. We search for 7 parameters: the PD parameters for the hip, thigh, and calf joints, as well as the body mass. Unlike the sim-to-sim experiment, we do not include ground friction in our search to allow the robot to adapt to various ground types it might encounter. We train the initial policy using UDR in simulation and use it to collect real-world data, sampling 2000 steps at a control frequency of 50Hz, which provides about **40 seconds of data**. Using these real-world samples, we then employ EASI for parameter search.

First, we demonstrate that after parameter search with EASI, the simulator became more realistic. In this experiment, we use the same initial policy and speed command (v=1.4m/s) to control the robot's movement in both sim and real environments. We plot the frequency spectrum of the robot's joint movements in **Fig.1 (a) of the Supp. PDF**. Despite being controlled by the same policy, there are significant differences in the joint movement frequencies between the origin simulator and real environments. After parameter search with EASI, the differences in the joint movement frequency spectrum are significantly reduced. 

Subsequently, we test the real robot's ability to follow speed commands. The results in **Fig.1 (b)** show that policies trained with EASI parameters have better speed tracking capabilities compared to those trained with origin parameters. This also proves that policies trained in the EASI-optimized sim environment can perform better in the real world.

`Q3. If the number of parameters to search increases, what will happen to EASI?`

`A3.` To explore the performance of EASI as the number of simulation parameters increases, we conducted a sim-to-sim experiment with the Go2 environment, searching for 25 parameters. In the experiment, we searched for the PD parameters of all motors in the Go2 robot and the mass of the robot. The results of the parameter search are shown in **Fig. 2** 

`Q4. More baseline methods to compare.`

`A4.` To better compare with existing sim-to-real algorithms, we add FineTune and GARAT [1] as baselines. 

In FineTune, we use the same demonstration data as EASI to fine-tune the UDR policy, allowing the UDR policy to adapt to the real-world environment based on real demonstration.

GARAT, a member of the GAT algorithm family, utilizes a GAN method to train an action transition network. This network adjusts actions in simulation such that, after adjustment, the actions applied in the sim environment result in state transitions more similar to those in the real demos. During the training of GARAT, we also used the same demonstration data as EASI.

**Table 1 in Supp. PDF** shows the performance in target environment of different algorithms after training in simulation. **All the results demonstrate the advantages of the proposed method.**

[1] Desai, Siddharth, et al. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems 33 (2020): 3917-3929.

`Q5. Hyperparameter sensitivity analysis for EASI.`

`A5.` Since EASI uses Evolutionary Strategies (ES) as the simulation parameter generator, it is much more stable in the training process compared to the original GAN algorithm and less sensitive to hyperparameters. In EASI, we select two hyperparameters for sensitivity analysis in **Table 2**，the evaluation metric is the L2 error between the searched parameter and the target parameters.

We first analyze the **hyperparameter** $\mu/\lambda$. In each evolution, we test $\lambda$ individuals and select the best-performing $\mu$ elite individuals for recombination and mutation. Generally, a higher $\mu/\lambda$ value helps maintain diversity in the population but may slow down the evolution process. Conversely, if $\mu/\lambda$ is too low, a lack of diversity can lead to local optima. To enhance parameter search, we recommend using a larger $\mu/\lambda$ and increasing the number of evolution steps, which will improve EASI's performance but also increase computational time.

Next, we analyze another **hyperparameter epoch_disc**, which refers to the number of times the discriminator is trained before each generation of evolution. Insufficient training may prevent the discriminator from achieving optimal performance, while excessive training can cause overfitting. In our experiments, we found that varying epoch_disc had only a minor effect on the final search results.

`Q6. Code is not available.`

`A6.` We have uploaded all the [codes](https://anonymous.4open.science/r/EASI-0998/) and supplementary materials.

`Q7. Minor errors.`

`A7.` We have corrected minor errors in the paper and have incorporated the suggested changes.

Please let us know if there are any remaining questions!

Meta-review: ## Recommendation

I recommend "Accept (poster)".

## Summary

The manuscript presents a novel method for simulator identification for sim2real robot learning. The authors combine real-world demonstrations, Generative Adversarial Networks (GANs) and Evolutionary Strategies (ES) in order to learn a distribution of simulation parameters. In particular, using the demonstrations they learn a discriminator (using GANs), which is then used as an objective function to an ES process. The final outcome of the EASI algorithm is a distribution of simulation parameters, and then normal uniform domain randomization (UDR) is performed (although one can use the learnt distribution in other ways). Overall, the paper is well-written, solves an interesting problem, and extensive experimentation/discussion is provided (including real-world experiments).

## Concerns

My main concern (and thus recommending poster instead of spotlight) is the impact of the contribution of the proposed method. In particular, the work is very close (in spirit) to {1} ([8] in the original manuscript) and in my opinion {1} possess a stronger main contribution. To further expand on this, the method in {1} is a closed-loop algorithm that learns in the real-world while learning the "optimal" simulation parameters. This is extremely important as the underlying assumptions about the system are less strong. On the contrary, EASI only learns using a "static" dataset (static is in quotes, since nobody constrains us of using the EASI proposed GAN-ES pipeline online; but the authors did not do this), and then we need to use this parameters to learn a policy. Given the above comments/realization, I am not sure about the potential impact of the contribution.

I expect the authors to include a paragraph to discuss the above comments/limitation (possibly in Section 6).

## Verdict

Despite my concern, I recommend acceptance (poster) for the following reasons:

1) The paper is well-written and clearly articulates the goals, solutions and discusses the results/assumptions
2) The idea of using a discriminator to learn an objective function for a black-box optimization procedure is straightforward (not saying this in a bad way; I like straightforward!), and very interesting!
3) The authors' rebuttals were clear, concise and provided interesting discussions or new experiments. Overall, the authors' comments showcase deep understanding of the literature.
4) All reviewers were positive about the work.

## References

{1}: Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N. and Fox, D., 2019, May. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 8973-8979). IEEE.
Decision: Accept (poster)