EASI: Evolutionary Adversarial Simulator Identification for Sim-to-Real Transfer

Authors: haoyu  dong
huiqiao  fu
wentao  xu
zhehao  zhou
chunlin  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:06:13

Abstract: Reinforcement Learning (RL) controllers have demonstrated remarkable performance in complex robot control tasks. However, the presence of reality gap often leads to poor performance when deploying policies trained in simulation directly onto real robots. Previous sim-to-real algorithms like Domain Randomization (DR) requires domain-specific expertise and suffers from issues such as reduced control performance and high training costs. In this work, we introduce Evolutionary Adversarial Simulator Identification (EASI), a novel approach that combines Generative Adversarial Network (GAN) and Evolutionary Strategy (ES) to address sim-to-real challenges. Specifically, we consider the problem of sim-to-real as a search problem, where ES acts as a generator in adversarial competition with a neural network discriminator, aiming to find physical parameter distributions that make the state transitions between simulation and reality as similar as possible. The discriminator serves as the fitness function, guiding the evolution of the physical parameter distributions. EASI features simplicity, low cost, and high fidelity, enabling the construction of a more realistic simulator with minimal requirements for real-world data, thus aiding in transferring simulated-trained policies to the real world. We demonstrate the performance of EASI in both sim-to-sim and sim-to-real tasks, showing superior performance compared to existing sim-to-real algorithms.

Main paper: 
 introduction: 
 With the development of Reinforcement Learning (RL) , an increasing number of RL controllers have demonstrated remarkable performance in complex, high-dimensional robot control tasks [1, 2] . Such RL algorithms require minimal domain-specific knowledge, offering superior control performance and more natural robot locomotion compared to traditional control methods. However, training these RL controllers requires plenty of trial and error processes, it's unsafe and costly to complete training tasks in the real world. On the other hand, simulation provides a cheaper and safer way to obtain a vast amount of trial data which is more suitable for RL training. Yet, due to simulation inaccuracies, disparities between simulation and reality inevitably exist and simulated-trained policies transferring directly to the real environment often result in poor performance, which is also known as 'reality gap'.To bridge the gap between simulation and reality, many studies focus on sim-to-real transfers to translate simulated-trained policy to reality. As one of the most popular sim-to-real methods, Domain Randomization (DR) [3] [4] [5] [6] injects random noise into observations, actions, and physical parameters during simulated training, which helps agents adapt to different environments, making the policies more robust. However, DR requires a good understanding of the specific domain to determine the appropriate ranges for these random variations. If these ranges are set inappropriately, it can hinder the successful transfer of policies. Additionally, DR often needs longer training time to ensure adaptability of the policy to various environments, and always results in suboptimal and conservative performance [6] . To tackle these challenges, recent attempts have been made to use real-world data for fostering policy transfer to real-world applications. One approach is optimizing simulator parameters to make the simulation more realistic [7] [8] [9] [10] [11] [12] [13] [14] . Unfortunately, these methods still face challenges, such as the need to iteratively collect large amounts of high-cost real-world data, difficulty in applying to high-dimensional control tasks, and instability in the parameter optimization process.In this work, we introduce Evolutionary Adversarial Simulator Identification (EASI), a novel sim-toreal method that frames sim-to-real transfer as a search problem and employs a unique collaboration between Generative Adversarial Network (GAN) and Evolutionary Strategy (ES). EASI searches for physical parameters to align state transitions between simulation and reality as closely as possible, ensuring that policies trained in simulation seamlessly transfer to the real world. Within EASI, ES acts as a generator in an adversarial competition with a neural network discriminator, which distinguishes state transitions originating from either simulation or reality. The discriminator assigns higher scores to state transitions more likely sampled from reality, acting as the fitness function for ES and guiding the evolution of physical parameter distributions. Concurrently, the discriminator's ability to differentiate between simulated and real state transitions improves through iterative training. EASI features simplicity, low cost, and high fidelity, requiring only a small amount of real-world data to bridge the gap between simulation and reality.In the experiments, we test EASI on 4 sim-to-sim tasks (Go2, Cartpole, Ant, Ballbalance) and 2 sim-to-real tasks (Go2, Cartpole). We first use Uniform Domain Randomization (UDR) to train a rough initial policy over a wide parameter range, allowing the agent to collect data in the target environment controlled by the initial policy. Then, we employ EASI for simulator identification that makes the simulator most similar to the target domain. The final policy is trained based on these optimized simulator parameters. Our experiments demonstrate that EASI can stably and rapidly search for simulator parameters, and the policies trained within the optimized parameter distribution can effectively transfer to the target domain. Video and code are shown at our page. In particular, we make three main contributions in this work:1. We propose a novel method, EASI, which employs a unique collaboration between GAN and ES to address sim-to-real challenges. 2. We model sim-to-real transfer as a search problem, searching for simulator parameters in an adversarial process to make state transitions between simulation and reality most similar. 3. We demonstrate the performance of EASI in both sim-to-sim and sim-to-real tasks, showing superior performance compared to existing sim-to-real algorithms.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly state the main contributions, including the key method and scope of the research, which are supported by the theoretical and experimental results presented in the paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Transferring simulated-trained policies to the real world in a stable and cost-effective manner has long been a goal for sim-to-real transfers. Previous research has approached sim-to-real transfer from both zero-shot and few-shot perspectives.Zero-shot sim-to-real transfer directly trains a policy capable of operating in reality without using any real-world data. Domain Randomization (DR) is one of the most intuitive and widely applied zero-shot sim-to-real methods [3] [4] [5] [6] . DR injects random noise into state observations, actions, and physical parameters to enhance the robustness of the policy, allowing the agent to adapt to various environments. However, DR requires specific domain prior knowledge and hand-engineering to determine the distribution of injected random noise, while also suffering from reduced control performance and high training costs. To improve DR, Akkaya et al. [15] used task performance as a metric to develop a curriculum that gradually increases the level of randomization. Mehta et al. [16] employed GAN to identify the most informative parts of the randomization parameter space, focusing more attention on these areas. Furthermore, some zero shot methods [17] [18] [19] introduced an adaptation module or estimator to predict robot state latent variables or explicitly estimate privileged information that the robot cannot access in reality, allowing the robot to adapt to different reality environments relying only on limited proprioception.Few-shot sim-to-real transfer leverages real-world data to facilitate policy transfer. Domain adaptation is one of the approaches that allows policy to generalize to reality. Image adaptation methods [20] [21] [22] [23] used GAN to transform simulated RGB images in a way that is matched to the reality. Grounded Action Transformation (GAT) [24] [25] [26] suggested action transformation that applying the transformed actions in simulation has the same effects as applying the original actions had on reality. In robotic tasks, domain adaptation methods often require a substantial amount of costly real-world data for effective transfer. Inspired by classic system identification [27] [28] [29] , some efforts focused on optimizing simulators so that the policies trained within them will better adapt to reality. Bayesian optimization was used to find the posterior distribution of simulator parameters based on real-world observations [13, 14, 30] . Chebotar et al. [8] and Chang et al. [31] treated the trajectory mismatch as the cost function and iteratively updated the simulator parameters. Memmel et al. [11] and Marija et al. [32] designed effective real-world exploration policies to sample more informative trajectories for system identification. TuneNet [7] employed supervised learning to train a neural network mapping simulation trajectories to parameter gradients toward the real world. Some approaches [9, 33] used RL to find proper parameters, treating trajectories as RL states and parameters as RL actions. SimGAN [10] employed GAN to distinguish between source and target domain dynamics and enhance the hybrid simulator to make it more realistic. However, most current simulation parameter optimization methods still suffer from instability, high cost of reality data collecting and are only suitable for low-dimensional control tasks.In this work, EASI employs GAN to differentiate between state transitions in simulation and reality, using ES to search for parameters that align the simulation more closely with the real world based on the GAN's distinction results. EASI requires lower demands on real-world data while enabling faster and more stable searching of simulator parameters. This helps us obtain high-fidelity, low-cost simulators to provide a real world like environment for RL training.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper includes a dedicated "Limitations" section where we discuss the potential weaknesses of our approach. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
domain randomization: 
 Consider a RL problem defined by the Markov Decision Process (MDP) represented by a tuple ⟨S, A, R, γ, P(ξ)⟩ , where S is the state space, A is the action space, R : S × A → R represents the reward function, γ is the discount factor, P(ξ) : S × A × S → R + represents the system's statetransition probability function under simulator parameters ξ. The state transition varies depending on the type of system and its specific parameters. We denote the real-world state transition as P r (ξ real ) and the simulation state transition as P s (ξ sim ). In RL process, an agent interacts with the environment and learns a policy π(a|s) to maximize the expected discounted return E p(τ |π)T -1 t=0 γ t R (s t , a t ) , where p(τ |π) = p(s 0 )T -1 t=0 p (s t+1 |s t , a t ) π(a t |s t ) represents the likelihood of a trajectory τ = {s 0 , a 0 , r 0 , s 1 , . . . , s T -1 , a T -1 , r T -1 , s T } under π. Due to the existence of the reality gap, policies trained in simulation often perform poorly when directly transferred to the real environment, and the manifestation of the reality gap in MDP lies in the difference between P r (ξ real ) and P s (ξ sim ). To address this issue, DR samples ξ sim ∼ Ξ ∈ R N and sets the simulator's parameters to ξ sim at each simulation training round, allowing the RL-trained policy to adapt to various P s (ξ sim ), thus improving system robustness. However, to train policies that perform well in the real world using DR, it is necessary to ensure that there exists ξ * ∈ Ξ such that P r (ξ real ) ≈ P s (ξ * ), which means it needs specific domain knowledge to determine Ξ.Evolutionary Strategy (ES) is a method that mimics biological evolution to solve high-dimensional, continuous-value domain problems. ES algorithm addresses the following search problem: maximize a nonlinear fitness function that is a mapping from search space R d to fitness value R. In each iteration (generation), the individual from the previous generation undergoes recombination and mutation to reproduce the next generation of individuals. Each offspring is evaluated through the fitness function, and individuals with the higher scores form the next generation's population.In this work, we use the discriminator as the fitness function of ES to evaluate the similarity of state transition, guiding ES to find ξ sim that deceives the discriminator to minimize the distance between P r (ξ real ) and P s (ξ sim ). GAN [34] consists two neural networks, generator G and discriminator D, engaged in an adversarial process to generate new data with the same statistics as the training set. G and D play a two-player minimax game with value function V (G, D), which can be described asEQUATIONwhere p(z) is the prior on input noise variables, p data is the training set. G and D continuously adjust parameters, with the ultimate goal of training a generator strong enough to let the discriminator unable to distinguish whether the outputs of the generator network are generated or sampled from training set.4 Evolutionary Adversarial Simulator IdentificationQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We are in the process of organizing the experiment code and we will provide open access to well-documented code.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
imitate state transitions: 
 Modeling the processes in reality and simulation as MDPs, the primary difference between the two MDPs lies in their state transitions P r (ξ real ) and P s (ξ sim ). In this paper, we focus on minimizing the state transition gap between simulation and reality. For simulation with variable physical parameters, we aim to find the distribution of the parameters to achieveEQUATIONIn EASI, we use a discriminator to distinguish whether state transitions come from simulation or reality, the discriminator D(s, a, s ′ ) can be trained usingEQUATIONwhere d M (s, a, s ′ ) and d B (s, a, s ′ ) denote the distribution of action a causing state transition from state s to s ′ in the demonstration dataset and in simulation respectively. As demonstrated in previous work [34] , the discriminator serves as a measure of the Jensen-Shannon divergence between P r (ξ real ) and P s (ξ sim ).It is important to acknowledge that, due to inherent design factors in the simulator, achieving state transitions that are precisely identical to those in the real world may be impossible. Nonetheless, our experiments have shown that EASI significantly enhances the simulator's realism, allowing policies trained in simulation to transfer more effectively to real world.Input: Initial parameter distribution Ξ (0) , motion control policy π 0 , demonstration state transition M, simulation state transition buffer B Output: Calculate discriminator reward rParameter distribution Ξ * 1 for generation i = 0, 1, 2, • • • , G do 2 Sample individuals by ξ (i) j ∼ Ξ (i) j = 1, 2, 3, • • • , N 3 for simulation environment j = 1, 2, 3, • • • , N do 4 Set simulator parameters to ξ (i) j 5 Use π 0 to sample trajectory τ j 6 Store τ j in B 7 end 8 for update step = 1, 2, 3, • • • , n do 9 b M = sample batch of K state transitions{(s k , a k , s ′ k )} K k=1 from M 10 b B = sample batch of K state transitions{(s k , a k , s ′ k )} K k=1 from BEQUATIONUsing ES to find next generation distributionΞ (i+1) = ES (ξ (i) , r (i) ) 16 endThe standard GAN discriminator objective detailed in Equation 3 typically uses a binary cross-entropy loss function which tends to encounter the problem of vanishing gradients due to saturation regions [35] . Specifically, if the discriminator is overly powerful, it may excessively classify state transitions, leading to identical bad rewards for all transitions sampled from the simulation. This makes it difficult to identify which parameters yield more realistic state transitions. Additionally, due to inherent discrepancies between the simulator and reality, the state transitions generated by the simulator may not perfectly align with those in the real world, further complicating the challenges posed by a strong discriminator. To address such challenges, WGAN-style discriminator [36] is proposed to mitigate the issue of gradient vanishing and has proven effective for robot-related tasks [2] . In our work, we employ Wasserstein loss to update the discriminator usingEQUATIONThis WGAN-style discriminators is an efficient approximation to the Earth-Mover distance which more effectively and smoothly measures the distance between two probability distributions. We clip the network weights to [-0.01, 0.01] as the original WGAN did to ensure the Lipschitz continuity of D(s, a, s ′ ) in the training process. Using Wasserstein loss, we could better evaluate the similarity between simulation and real world state transitions.Our goal is to find a parameter distribution (e.g. Gaussian distribution) that makes the simulator most similar to the demonstration. The distance between P r (ξ real ) and P s (ξ sim ) is measured by D(s, a, s ′ ) and what we need to do is to find the best parameter distribution Ξ * that get the highest reward of D(s, a, s ′ ), which could be described asEQUATIONIn EASI, we employ the ES as the generator, with the goal of creating a parameter distribution for the simulator that closely mimics the state transitions observed in the demonstration data. EASI uses the ES to select parameters that receive higher rewards from the discriminator, designating these parameters as elites. The elites then undergo recombination and mutation to produce the next generation of parameters. The discriminator evaluates whether the state transitions in the generated trajectories are sufficiently similar to those of the expert demonstrations, assigning scores based on the degree of similarity. Simultaneously, the discriminator is trained using the trajectories collected during simulation and demonstration, guided by Equation 4. With the selection process driven by the ES, the state transitions produced by the simulator increasingly resemble those in the demonstration data, and the discriminator can better distinguish the gap between simulated and demonstration state transitions. Ultimately, this algorithm yields a set of simulator parameters that align the state transitions in the simulated environment with those in the real world. This optimized parameter set is then utilized to train policies that can be directly applied in real world. The schematic overview of the EASI architecture is shown in Fig. 1 , and the pseudo-code is shown in Algorithm 1.It's notable that when sampling trajectory from simulation and real world, the motion control policy we use does not necessarily match the final desired policy and the robot may be controlled by MPC [37] , PID [38] , RL, or even manually operated by humans, as long as it can collect state transition containing enough state transition information we need. This characteristic of our algorithm significantly reduces the cost of collecting real data. With our algorithm, using a rough policy run in the real world to collect suboptimal trajectories is enough to find a proper parameter for the simulator. 
experiment: 
 In this section, we conduct experiments in various tasks to validate the performance of EASI and answer three questions:1. Can EASI enhance the simulator's similarity to real-world?2. Can EASI improve performance in sim-to-real transfer tasks?3. How much real-world data is required for EASI?Task Setup We test EASI in 4 sim-to-sim tasks illustrated in Fig. 2 and 2 sim-to-real tasks presented in Fig. 3 . We utilize Isaac Gym [39] as the simulator. In the simulator identification process, we create 300 parallel environments for EASI to evaluate the similarity between reality and simulator with different parameters. For Cartpole, Ant, and Ballbalance, we use SAC [40] to train a policy (4 layer MLP) as the locomotion controller. As for Go2, we employ Ess-InfoGAIL [41] to train a quadruped controller.Baselines (1) Uniform Domain Randomization (UDR): Uniformly sampling parameters from Ξ U DR at the beginning of each training iteration. (2) Oracle: Directly training in the pseudo-real environment in sim-to-sim experiments, representing the ideal upper bound for sim-to-sim transfer tasks.(3) FineTune: Fine-tune the UDR policy using real-world data, enabling the UDR to adapt to the real world in advance to improve transfer performance. (4) GARAT: GARAT [26] adjusts actions from the policy such that, after adjustment, the actions applied in the simulator result in state transitions more similar to those in the real demonstration.EASI Implementation There are no specific requirements for the selection of the ES in EASI, mainstream ES such as CMA-ES [42] , NES [43] etc. can be used. In this paper, we employ (µ/µ I , λ)-ES [44] as the generator with the setting µ = 150 and λ = 300, which means every generation has 300 individuals and 150 elites are chosen to undergo recombination and mutation to generate the next generation. For the collection of the demonstration dataset, we use UDR to train a rough policy, and then use the policy to control agents in the target domain collecting trajectories.EASI can quickly accomplish parameter searching tasks. The discriminator can directly evaluate the similarity between simulator and reality by leveraging a single trajectory sampled from the simulator with a specific parameter. Furthermore, in Isaac Gym, we can parallelly collect trajectories in hundreds of environments with different parameters which means we could evaluate hundreds of parameters parallelly. In our experiment, running on a PC equipped with Intel i5-13600KF and RTX 4060 Ti, EASI completed the evolutionary adversarial searching process in less than 10 minutes. To validate the theoretical capability of EASI for simulator identification, we conduct simto-sim experiments in 4 tasks (Cartpole, Go2, Ant, and Ballbalance). Initially, a set of parameters ξ r is selected as the target parameters for the simulator, which is regarded as the pseudo-real environment. For UDR, we useΞ U DR = U [ 1 k × ξ r , k × ξ r ]as the uniform parameter distribution and train an initial policy π 0 with UDR. Depending on the difficulty of the task, we set k = 3 for the Cartpole, Ant, and Ballbalance tasks, and k = 2 for Go2 task. Then π 0 is used to collect 200 trajectories in the pseudo-real environment as demonstration state transition M. During the EASI parameter optimization process, two approaches are used for selecting the initial parameter distribution: Within Distribution (WD) and Out of Distribution (OOD). In the WD experiment, the initial parameter distribution Ξ (0) is consistent with the domain randomization distribution Ξ U DR , and the target parameter ξ r is included within the initial parameter distribution Ξ (0) . In the OOD experiment, we deliberately set incorrect initial parameter distribution Ξ (0) (e.g. set as U [0, ξ r ] or U [ξ r , 2 × ξ r ]) so that the target parameter is outside the initial parameter distribution. Taking Go2 and Ant environment as examples, there are 8 parameters for Go2 and 11 parameters for Ant to identify. Here we choose 2 of the parameters in each task to demonstrate the generating process of the parameter. Fig. 4 represents the convergence process of parameters, where the x-axis represents the generations of ES, and the y-axis represents the parameter values. It can be observed from the figure that at the beginning, parameters are uniformly distributed within Ξ (0) . As the parameter evolution progresses, the parameter distribution quickly adjusts to the vicinity of the target parameters. This experiment showcases EASI's capability to adjust parameter distributions, even when the initial After parameter optimization, we utilize the converged parameter distribution Ξ * to train new policies. In each task, EASI runs three times with different random seeds to avoid randomness. Figure 5 depicts the policy's performance in the pseudo-real environment throughout the training process. The X-axis represents the number of optimization steps for the policy during training in the simulation, and the Y-axis indicates the policy's performance when tested in the pseudo-real environment. In all tasks, our method outperforms UDR and even demonstrates comparable performance to the policy trained directly in the pseudo-real environment.Subsequently, we conduct a test to evaluate the performance of EASI across various target environments. We fix the initial parameter distributionΞ (0) = U [ 1 k × ξ r , k × ξ r ]and adjust one of the parameters ξ i r to 1/4, 1/2, 2, and 4 times its original value, denoted as ξ ′ r . We choose the proportional parameter of the PD controller for the Cart DoF in Cartpole, the proportional parameter of the PD controller for the Thigh Dof in Go2, the body mass of Ant, and ball mass of Ballbalance as the parameters to be adjusted. We use the simulation with parameter ξ ′ r as the target environment and utilize EASI to identify. The performance of trained policies in different target environments are shown in Fig. 6 . It can be seen that the policies trained with UDR are overly conservative or fail to adapt to the target environment due to the gap between source and target environments. On the contrary, simulators optimized by EASI are more similar to the target environment, thus policies trained in these simulations are more adaptable to target environments and achieve higher performance in the target domain. In specific scenarios, we compare EASI with other sim-to-real methods, with detailed results presented in Table 1 . The experimental results demonstrate that policies trained with EASI consistently achieve better performance in the target environment, highlighting EASI's effectiveness in facilitating a seamless transfer of policies to the target environment.Cartpole In real-world control, using Cartpole as an example, we run the policy on a PC receiving state information from robot and sending action commands to the robot. The control frequency for the Cartpole task is 50Hz. We evaluate the performance of the policy by the angle error of pole and the velocity of cart, which reflect the accuracy and stability of the control policy. We train an initial policy using UDR and sample 50 trajectories including approximately 10000 state transitions in real-world environment. For EASI, initial parameter distribution and other algorithm settings remain the same as sim-to-sim tasks. Table 2 presents the performance of different algorithms in real-world Cartpole tasks, and it can be seen that the policies trained in the simulator optimized by EASI achieve more accurate and stable control in the Cartpole task and show significant improvements compared to previous sim-to-real method.To further evaluate the performance of EASI in real-world scenarios, we conduct an experiment using the Unitree Go2 quadruped robot. We search for 7 parameters: the PD parameters for the hip, thigh, and calf joints, as well as the body mass. Unlike the sim-to-sim experiment, we do not include ground friction in our search to allow the robot to adapt to various ground types it might encounter. We train the initial policy using UDR in simulation and use the initial policy to collect real-world data, sampling 2000 steps at a control frequency of 50Hz, which provides about 40 seconds of data. Using these real-world samples, we then employ EASI for parameter search. Figure 7 : (a) Using the same policy, there is a significant difference in the motion spectrum of the Go2's joints between the simulation and the real environment due to the reality gap. After optimizing with EASI, the motion spectrum of the Go2's joints in the simulation becomes closer to that in the real environment. (b) Comparison of speed tracking in real environments between policies trained with EASI parameter distribution and trained with UDR parameter distribution.After parameter search with EASI, the simulator become more realistic. In this experiment, we use the same initial policy and speed command (v=1.4m/s) to control the robot's movement in both simulation and reality environments and plot the frequency spectrum of the robot's joint movements in Fig. 7(a) . It is notable that the final output of EASI is the distribution of simulator parameters Ξ * . To elucidate the differences in simulation with optimized parameter distribution Ξ * and original parameter distribution Ξ (0) (consistent with Ξ U DR ), we utilize the means of these distributions ξ * and ξ (0) as the parameters for the simulator, referred to as sim-EASI and sim-UDR, respectively. Despite being controlled by the same policy, there are significant differences in the joint movement frequencies between the sim-UDR and real environments, indicating a substantial reality gap between simulator and the real world. After parameter optimization with EASI, the differences in the joint movement frequency spectrum have been significantly reduced, indicating that the sim-EASI more closely resembles real-world. Subsequently, the real robot's ability to follow speed commands is tested. Using the parameter distribution Ξ * obtained from EASI as the simulator parameter distribution to retrain a policy before testing on the real quadruped robot. During the testing phase, we evaluate the robot's performance in the same scenarios using various speed commands. The results in Fig. 7 (b) show that policies trained with EASI parameters have better speed tracking capabilities compared to those trained with origin parameters. This also proves that policies trained in the EASI-optimized simulator can perform better in the real world.We design a data budget experiment based on sim-to-sim experiments to test the impact of demonstration data quantity on EASI. In this experiment, we limit the number of trajectories sampled in the target domain to 100, 50, and 1 respectively, and conduct EASI using the limited trajectories. The experimental results of the data budget are presented in Table 3 . The results demonstrate EASI's data efficiency, as only a small amount of real-world data is needed for EASI to identify parameters that align the simulation closely with the target environment and enhance the policy's adaptability in the real world. 
limitations: 
 Even though EASI requires relatively low amounts of real-world data, it still necessitates collecting corresponding state transition data for each specific scenario and maintaining certain standards regarding sensor error ranges. When sensors exhibit significant noise, additional techniques for imitation learning from imperfect demonstrations are needed. Furthermore, even within the same scenario, substantial differences in task styles can reduce EASI's performance. To achieve optimal results, it is recommended to collect expert data separately for each task. 
conclusion: 
 In this work, we introduce EASI, a novel sim-to-real method that combines GAN with ES. EASI uses ES as a generator in an adversarial competition with a neural network discriminator to find physical parameter distributions that align state transitions between simulation and reality. The discriminator serves as the fitness function for ES, guiding the evolution of parameter distributions. We conducted a series of experiments to validate EASI's effectiveness and data efficiency in both sim-to-sim and sim-to-real tasks, demonstrating its exceptional performance. We believe EASI is a general sim-to-real method characterized by simplicity, low cost, and high fidelity. In future work, we will evaluate EASI's performance on a broader range of robotic platforms and industrial scenarios.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Error bars are reported for key results. All figures and tables include error bars capturing the variability due to different random seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper specifies the type of compute resources used, including CPU and GPU specifications, etc. These details are provided in Section 5.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The potential societal impacts, both positive and negative, are discussed in the Appendix C.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The method proposed in this paper is mainly applied to robot control tasks and poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All external assets used in this research are properly credited, and their licenses and terms of use are respected. This information is provided in the references and acknowledgment sections.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: New assets, including datasets and code, are thoroughly documented, and this documentation is provided alongside the assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To explore the performance of EASI as the number of simulation parameters increases, we conducted a sim-to-sim experiment with the Go2 environment, searching for 25 parameters. In the experiment, we searched for the PD parameters of all motors in the Go2 robot and the mass of the robot. The results of the parameter search are shown in Fig. 8 (Due to space limitations, 10 out of 25 results are displayed). Since EASI uses Evolutionary Strategies (ES) as the simulation parameter generator, it is much more stable in the training process compared to the original GAN algorithm and less sensitive to hyperparameters. In EASI, we select two hyperparameters for sensitivity analysis in Table 4 and Table 5 ,the evaluation metric is the L2 error between the searched parameter and the target parameters.We first analyze the hyperparameter µ/λ. In each evolution, we test λ individuals and select the best-performing µ elite individuals for recombination and mutation. Generally, a higher µ/λ value helps maintain diversity in the population but may slow down the evolution process. Conversely, if µ/λ is too low, a lack of diversity can lead to local optima. To enhance parameter search, we recommend using a larger µ/λ and increasing the number of evolution steps, which will improve EASI's performance but also increase computational time.Next, we analyze another hyperparameter epoch_disc, which refers to the number of times the discriminator is trained before each generation of evolution. Insufficient training may prevent the discriminator from achieving optimal performance, while excessive training can cause overfitting. In our experiments, we found that varying epoch_disc had only a minor effect on the final search results. In our work, we have proposed EASI for simulation trained strategy transfer to reality. This could potentially drive the application of robotics technology in the real world, thereby positively impacting societal operational efficiency. Sim-to-real techniques reduce the need for extensive physical trials, minimizing costs associated with hardware testing and prototyping. This can accelerate the development of robotic technologies, making them more accessible to a broader range of industries.However, there is a risk that developers may become overly reliant on simulation results, potentially leading to systems that perform well in simulated environments but fail in the real world due to unmodeled complexities. Additionally, it is important to recognize that the capabilities of simulations have limits, and we cannot expect simulations to perfectly mirror reality. Resources should still be invested in designing better robot structures and developing more advanced simulators.Moreover, there are growing concerns about the computational resources required for current training schemes. The increasing demand for computational power undoubtedly consumes significant amounts of energy and resources. We are continually exploring more efficient algorithms to address this challenge.Justification: The research adheres to the NeurIPS Code of Ethics, with careful consideration given to ethical standards throughout the study.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: ## Recommendation

I recommend "Accept (poster)".

## Summary

The manuscript presents a novel method for simulator identification for sim2real robot learning. The authors combine real-world demonstrations, Generative Adversarial Networks (GANs) and Evolutionary Strategies (ES) in order to learn a distribution of simulation parameters. In particular, using the demonstrations they learn a discriminator (using GANs), which is then used as an objective function to an ES process. The final outcome of the EASI algorithm is a distribution of simulation parameters, and then normal uniform domain randomization (UDR) is performed (although one can use the learnt distribution in other ways). Overall, the paper is well-written, solves an interesting problem, and extensive experimentation/discussion is provided (including real-world experiments).

## Concerns

My main concern (and thus recommending poster instead of spotlight) is the impact of the contribution of the proposed method. In particular, the work is very close (in spirit) to {1} ([8] in the original manuscript) and in my opinion {1} possess a stronger main contribution. To further expand on this, the method in {1} is a closed-loop algorithm that learns in the real-world while learning the "optimal" simulation parameters. This is extremely important as the underlying assumptions about the system are less strong. On the contrary, EASI only learns using a "static" dataset (static is in quotes, since nobody constrains us of using the EASI proposed GAN-ES pipeline online; but the authors did not do this), and then we need to use this parameters to learn a policy. Given the above comments/realization, I am not sure about the potential impact of the contribution.

I expect the authors to include a paragraph to discuss the above comments/limitation (possibly in Section 6).

## Verdict

Despite my concern, I recommend acceptance (poster) for the following reasons:

1) The paper is well-written and clearly articulates the goals, solutions and discusses the results/assumptions
2) The idea of using a discriminator to learn an objective function for a black-box optimization procedure is straightforward (not saying this in a bad way; I like straightforward!), and very interesting!
3) The authors' rebuttals were clear, concise and provided interesting discussions or new experiments. Overall, the authors' comments showcase deep understanding of the literature.
4) All reviewers were positive about the work.

## References

{1}: Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N. and Fox, D., 2019, May. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 8973-8979). IEEE.

Reviews: review: paper_summary: paper_summary: The paper introduces Evolutionary Adversarial Simulator Identification (EASI), a novel sim-to-real method using a combination of Generative Adversarial Network (GAN) and Evolutionary Strategy to bridge the gap between simulation and reality. EASI optimizes simulator parameters by aligning state transitions between simulation and reality, enhancing policy transferability. The method is tested on various tasks, demonstrating superior performance and stability compared to existing sim-to-real algorithms through experiments.
strength_weakness: strength_weakness: 1) This paper lacks explanations on important technique details. Such as how the transition distribution is derived from the demonstration data. 
2) Some parts of the paper's demonstration are confusing, for example, if Fig.3 reflects the training of proposed methods in real env or simulator env. If it is trained in a real env, it doesn’t make sense for sim-to-real transfer.
3) The paper discussed comprehensive related work, but very few baseline methods were compared regarding the performance in the paper, such as Grounded Action Transformation, etc. 
4) The setting of the experiment is not as convincing as what is claimed when targeting the existing work’s problems. Such as line 42-43, other methods are hard to perform well in large-dimensional parameter space, this paper only conducted experiments on the environment with a maximum of 11 parameters, which might still be simple env. 
5) The paper claimed `yes` in checklist 5, open-access of code, but it is not available anywhere in the paper, which is not aligned with the checking claims, and may hinder the reproducibility.
questions: questions: 1) How do you capture/model the state transition, since the sim-transitions will be used to measure the gap with the real-transitions by JS?
2) In algorithm lines 9-10, Do they require the same initial state s_k? If yes, how does the work make sure they are aligned? If not,  how do the authors measure the transition divergence? I assume the JS divergence was employed (line 157), it would require two distributions, how do the authors obtain/estimate the real-transitions distribution? 
3) The baseline methods are only the Oracle and Uniform Domain Randomization(UDR), can authors incorporate the GAT branch methods, and Domain Adaptation methods as well? 
4) I have concerns about the generalizability of the proposed method since for GAN-based methods, it is notoriously difficult to train, especially for high-dimensional feature space, and what is presented in the paper seems still a relatively simple environment. 
5) Can authors explain If Fig.3 reflects the training of proposed methods in real env or simulator env? If it is trained in a real env, it doesn’t make sense for sim-to-real transfer.
limitations: limitations: 1) The paper lacks proof of the high-dimensional transfer performance either empirically or theoretically. 


2) And the work seems not able to easily adapt to new tasks, it requires training all over again (using the GAN paradigm) in a new real-world environment.
paper_summary: paper_summary: The paper tackles the issue of finding correct parametrizations for simulators for robot tasks in order to close the sim-to-real gap. To optimize the simulation parameters, evolutionary strategies (ES) are employed in combination with a discriminator function (trained in a GAN setting). The study shows that the presented approach yields better results (running in whatever setting is considered "real") than the baseline approach Uniform Domain Randomization (UDR).
strength_weakness: strength_weakness: Some connections within the paper are very confusing. Alg. 1 is not properly referenced und thus not discussed deeply enough. Fig. 1 is not referneced properly and comes way too late within the paper. Various explanations on GANs are confusing as -- in this approach -- we do not have a generator in its own right but build a slightly different architecture. The plots in Fig. 2, e.g., are explained at various non-continuous locations and thus very hard to follow.

The study lacks some stronger comparison, but if the field does not have much to offer here at the moment, this is acceptable. I am missing some disucssion about the comparative use of compute. EASI seams much more involved than UDR here.

The limitation section is very good to have, but quite weak as limitation sections go.

Eq. 4 is unclear to me. Does it contain a typo?

Several formal issues persist, including the usage of abbreviated forms ("it's"), various references without a qualifier ("according to 4"), and missing spaces. I recommend thorough proofreading. I also find it to be more common to use "an RL problem" and "an MDP".

Some more practical illustrations on the impact of the method would have been nice, especially the real world experiments.
questions: questions: Eq. 4 is unclear to me. Does it contain a typo?
limitations: limitations: The limitation section is very good to have, but quite weak as limitation sections go.
paper_summary: paper_summary: This submission presents an approach to sim2real transfer by predicting better simulation hyperparameters via Evolutionary Adversarial Simulator Identification (EASI). EASI is a combination of a generative adversarial network (GAN) and evolutionary strategy approach to generating simulation hyperparameters. Concretely however in the GAN setup, the generator is not another neural network, only the discriminator is a neural network that predicts whether a state transition $(s, a, s')$ is real or not. The generator is now an evolutionary search procedure that generates a distribution of simulation hyperparameters that strives to maximize discriminator rewards. Importantly EASI does not simply generate fixed simulation hyperparameters, but rather a range/set of hyperparameters and can be seen as a more "educated" domain randomization approach when used for training. The result of EASI is better sim2real transfer of policy performance when compared to a standard uniform domain randomization technique. Empirical tests are on robotics control tasks tested can match the oracle.
strength_weakness: strength_weakness: - Only control tasks are tested which generally are easier to solve with just domain randomization (albeit EASI does better here). It would be useful to see more complex tasks beyond simple control tasks that leverage manipulation as well. This point doesn't lower my score but if more complex tasks were analyzed I'd be happy to raise the score further to reflect the bigger impact of the method.
- To leverage EASI some sim+real data is required still to train the discriminator and run the evolutionary search. While the amount of real data in experiments can be kept to just one expert demonstration, it's not clear how much simulated data is needed, and what distribution of simulated data is needed. Section 5 mentions UDR is used to train a rough policy but how rough is rough? What if UDR does not get any useful data / what would be considered sufficient UDR to then kickstart EASI?
- Since EASI has to run UDR itself first to get some sim data to train on, overall it could be that running UDR for as long as it takes to run UDR + EASI achieves the same performance, although Figure 3 suggests otherwise. There don't seem to be figures/information about how long the first UDR is done for EASI. It might be fairer to run the UDR baseline for as many online steps EASI uses for training after EASI and before with the rough UDR.
- As pointed out in the limitations section, this sim2real setup requires measuring state in the real world and simulation. For control tasks where the state often is just the robot state and can be measured via sensors, this is fine. For more complex tasks involving manipulation of other objects, this approach will probably not work well.

Happy to raise my score if the above and questions are addressed
questions: questions: Questions:
- How come in ball balance the oracle method (SAC training on dense rewards with the correct sim parameters) performs worse than EASI, which has SAC training on sim parameters predicted by rough UDR.
- Are dense rewards used for all tasks?
- What if UDR does not get any successful demonstrations e.g. in a cartpole task say the pole is never swung up completely. It sounds like in this scenario EASI might not generate a good set of hyperparameters that would be important for unseen parts of the task?
Typos:
- What does "domain priority" in section mean? The line "However, DR needs specific domain priority and hand-engineer to determine the
79 distribution of injected random noise." might have typos and grammar issues
limitations: limitations: A limitation section is provided and adequately points out the assumption that EASI relies on expert trajectories and accurate state estimation of the same observation data in simulation and the real world.
paper_summary: paper_summary: This work studies the sim-to-real transfer using an evolution strategies with a learned discriminator. The discriminator learns to distinguish the state transition between real and simulation. The evolution strategies aim to optimize the parameters of the simulator so that the data generated by the simulator is undistinguishable by the discriminator. Experiments on sim-to-sim and sim-to-real cases have been conducted.
strength_weakness: strength_weakness: The details of the employed GAN architecture lacks furture description.
questions: questions: 1) It seems that the major technical contribution of this work is to learn an objective function for the ES using GAN, for simulator calibration or say parameter inference. Thus, I am very curious about the architecture of the used GAN and how are the tuple s, a, s' processed in the training. More descriptions on the details of used GAN are welcome.
2) In Figure 3, why the EASI can outperform the oracle that indicates the upperbound of the performance?
3) For sim-to-transfer case, the oracle may not represent the upperbound, in which case some advanced compared algorithms need to be compared in the experiments.
4) It is interesting to see EASI can adjust the parameters out of the initial range. What makes it capable of doing so?
5) How is the scalability of the EASI, I mean if the number of parameters increases, what will happen to EASI?
limitations: limitations: YES
paper_summary: paper_summary: The paper introduces an approach to address the challenges of transferring reinforcement learning (RL) policies from simulation to real-world applications. Traditional methods like Domain Randomization (DR) require significant domain-specific knowledge and extensive training time, which often result in suboptimal performance. EASI combines Generative Adversarial Networks (GAN) and Evolutionary Strategies (ES) to identify physical parameter distributions that align state transitions between simulation and reality, thus minimizing the reality gap.
strength_weakness: strength_weakness: #### Methodological Concerns 

1. **Wasserstein GAN loss**: The Wasserstein GAN loss was first introduced by the Wasserstein GAN. Using the Wasserstein GAN loss with clipped network weights in the range [-0.01, 0.01] does not seem to be a novel contribution by the author. 

  

#### Experimental Limitations 

1. **Diversity of Tasks**: While the paper presents experiments on various tasks, the real-world applications are somewhat limited. For instance, in Sim to Real experiments, only Cartpole is used. Expanding the experiments to include more complex real-world scenarios, such as robot dogs or navigation tasks, would strengthen the claims about EASI's broad applicability. 

  

#### Practical Considerations 

1. **Computational Resources**: The computational cost of EASI, especially for training GANs and performing evolutionary searches, is not thoroughly discussed. Providing insights into the computational requirements and potential optimizations would be valuable for practical deployment. 

2. **Scalability**: The scalability of EASI to high-dimensional state spaces and action spaces is not well-addressed. Including discussions or preliminary results on scaling the method to more complex systems would be beneficial.
questions: questions: 1. **Parameter Sensitivity and Selection**:  

   - **Question**: How sensitive is EASI to the choice of hyperparameters for both the GAN and the ES components?  

   - **Suggestion**: Provide a detailed sensitivity analysis or guidelines for selecting these parameters to help with replicability and optimization in different settings. 

   

2. **Diversity of Experimental Tasks**: 

   - **Question**: How does EASI perform across more complex real-world scenarios? 

   - **Suggestion**: Expand the experimental validation to include other real-world applications (for example, navigation) to strengthen claims about the method’s general applicability. 

   

3. **Computational Resources**: 

   - **Question**: What are the computational requirements for training GANs and performing evolutionary searches in EASI? 

   - **Suggestion**: Discuss the computational costs and potential optimizations to make the method more accessible for practical deployment. 

  

4. **Scalability**: 

   - **Question**: How scalable is EASI to high-dimensional state and action spaces? 

   - **Suggestion**: Address the scalability of the method and include preliminary discussions on scaling to more complex systems.
limitations: limitations: The limitations are thoroughly discussed in Section 6.

Comments: title: Response to Reviewer ZLtE
Official Comment by Reviewer ZLtE
Response to Reviewer zVKJ
Response
Response to Reviewer yoqm
Response to Reviewer ZLtE
Response to Reviewer zVKJ
Response to Reviewer rASM
Thanks for your clarification
Response
Official Comment by Reviewer ZLtE
comment: Your reactions to my notes were satisfactory, justifying the positive evaluation I gave originally.
Dear Reviewer ZLtE:

We greatly appreciate your time in reviewing our paper and reading the follow-up rebuttal! We're thrilled for your recognition of our work. Thank you very much!

Best regards,

The Authors
I also read the rebuttal of the authors with other reviewers. Thanks the authors for the detailed and effective responses. I raise my score to 6.
Dear Reviewer zVKJ:

Thank you for your suggestions. We are currently working hard to validate EASI's performance on more tasks, including pusher (completed), grasping, and dexterous manipulation. However, training to complete all of these tasks requires more time, making it difficult to provide final experimental results during the discussion. We will continue to refine these experiments and plan to include them in a future version of this paper. Additionally, we will continue to promote the application of EASI in more real industrial production scenarios. 

Again, sincerely thank you for your time and insights.
Thanks for the clarification. I'm glad you are considering a pushing task but I still would consider such a task just as easy (if not easier) than locomotive tasks even. It is hard to say as this is somewhat subjective. My bar for challenging manipulation is actual grasping (with gripper or dextrous hands) of at minimum a cube, but better to be e.g. YCB dataset objects.
Dear Reviewer 2SzS:

Thank you once again for your insightful comments and helpful suggestions and we're thrilled for your recognition of our work. As the deadline for author-reviewer discussions is approaching,  if you have any further questions or concerns, please let us know. Thank you very much for your time.
Dear Reviewer ZLtE:

Thank you once again for your insightful comments and helpful suggestions. As the deadline for author-reviewer discussions is approaching, if you have any further questions or concerns, please let us know. Thank you very much for your time.
Dear Reviewer yoqm:

Thank you for your detailed questions and suggestions，they have been very enlightening for us. If you have any further questions, please contact us.

Sincerely, thank you for your time.
Thanks for your rebuttal. My concerns have been resolved and I decide to improve my score.
Dear Reviewer ZLtE:

We sincerely appreciate your valuable suggestions for our work. We will incorporate some of the content you mentioned in future versions of the paper to make it more complete and solid.

Sincerely, thank you for your time.
Dear Reviewer zVKJ:

We also hope to validate EASI's capabilities in challenging manipulation tasks and we plan to test EASI's performance in the Mujoco Pusher environment—a task where a robotic arm pushes a small ball to a target location—during the Discussion Period, and we will provide you the results as soon as possible.

For Q7, when using the Isaac Gym GPU simulator running on a RTX 4060Ti, EASI actually runs quite quickly. We measured the time required for EASI in the most complex Go2 environment. In this experiment, we used 300 environments for evolution, meaning that each evolution step involves collecting 300 trajectories in 300 environments with different parameters. In each round of evolution, we measured the time required for each part:

|  process   | time consuming(s)  |
|  :------: | :------:  |
| sim trajectory collection  | 6 |
| discriminator training  | 0.09 |
| parameter evolution  | 0.11 |

Generally, parameter convergence can be achieved within 30 to 50 generations. The parameter search with EASI can be completed in less than 10 minutes. Compared to RL training times, which can take several hours or even days, EASI is already very fast. 

While using UDR directly can save the time spent on parameter search, as mentioned in the paper, we often lack the knowledge to set a reasonable parameter range for UDR. Without specialized knowledge to adjust the UDR parameter range, the trained policy might not transfer well to the real environment. Thus, directly using UDR may require extensive expertise and trial and error, which can be time-consuming compared to using EASI. 

Finally, your valuable and insightful feedback has been extremely helpful to us. The issues you raised have greatly help to our work. We sincerely thank you for your time and suggestions!
Dear Reviewer rASM:

Thank you for your constructive feedback and great efforts in helping us improve our work. 
We will continually improve the paper based on your recommendations. If you have any further concerns about the paper, please contact us.

Once again, thank you!
Thanks for your rebuttal. My concerns have been resolved and decided to improve my score.
Good to see the limitations are mentioned in the paper, I must have missed one or two of them.

All concerns except for Q7 are addressed. I think Q8 can't be addressed easily anyway but if somehow EASI can achieve that level of ability with harder manipulation tasks I would say this has a lot more impact than I currently think. Not to say it is not possible but without empirical results I cannot make a fair judgement here.

Regarding Q7, I understand that you ensure the same data is fed into EASI and UDR. However there are likely computing differences right? In EASI you are using the EASI setup + running UDR, meaning additional computation and potentially runs very slowly. This does not change my opinion however that EASI is much better. The figures suggest that UDR converges to some suboptimal results compared to EASI. Just pointing out some may think this is an issue and may request wall time results (which is not standard in RL but if you use GPU simulation it really should be since sample efficiency is always poor with high numbers of parallel environments).

I will raise my score to 7 as I think this is a technically and well done paper. I choose not to raise higher as there is still a big limitation of having to measure real world states and the benchmarking goes as far as just locomotive tasks (which have already been solved quite well via other methods like straight RL+reward tuning so I find this work less impactful).
Thank you for the detailed response. 
Actually, I like the idea of this paper. And all my previous concerns have been addressed. 
Very appreciated for your works!