REX: GPU-Accelerated Sim2Real Framework with Delay and Dynamics Estimation
Authors: Bas van der Heijden
Jens Kober
Robert Babuska
Laura Ferranti
Venue: Accepted by TMLR ()

Abstract: Sim2real, the transfer of control policies from simulation to the real world, is crucial for efficiently solving robotic tasks without the risks associated with real-world learning. However, discrepancies between simulated and real environments, especially due to unmodeled dynamics and latencies, significantly impact the performance of these transferred policies. In this paper, we address the challenges of sim2real transfer caused by latency and asynchronous dynamics in real-world robotic systems. Our approach involves developing a novel framework, REX (Robotic Environments with jaX), that uses a graph-based simulation model to incorporate latency effects while optimizing for parallelization on accelerator hardware. Our framework simulates the asynchronous, hierarchical nature of real-world systems, while simultaneously estimating system dynamics and delays from real-world data and implementing delay compensation strategies to minimize the sim2real gap. We validate our approach on two real-world systems, demonstrating its effectiveness in improving sim2real performance by accurately modeling both system dynamics and delays. Our results show that the proposed framework supports both accelerated simulation and real-time processing, making it valuable for robot learning.

Published in Transactions on Machine Learning Research (2/2025)
REX: GPU-Accelerated Sim2Real Framework with Delay and Dynamics Estimation
Bas van der Heijden
S.hetjden Shotmail. com
Cognitive Robotics
Delft University of Technology
Jens Kober
j.koberSudelff..r
Cognitive Robotics
Delft University of Technology
Robert Babuska
r. r.babusk
Cognitive Hobotics, Delft University of Technology, CIRCC Czech Technical University in Prague
Laura Ferranti
Lfernanti@tudelf..mm
Cognâtive Robotics
Delft University of Technology
Reviewed on OpenReview: https: // optpreveeerre... net/ forum? id= 04CQ5AM5yP
Abstract
Sim2real, the transfer of control policies from simulation to the real world, crucial for efficiently solving robotic tasks without the risks associated with real-world learning. How- ever, discrepancies between simulated and real environments, especially due unmodeled dynamics and latencies, significantly impact the performance of these transferred policies. In this paper, address the challenges of sim2real transfer caused by latency and asyn- chronous dynamics in real-world robotic systems. Our approach involves developing novel framework, REX (Robotic Environments with jaX), that uses a graph-based simulation model to incorporate latency effects while optimizing for parallelization on accelerator hard- ware. Our framework simulates the asynchronous, hierarchical nature of real- world systems, while simultancously estimating system dynamics and delays from real-world data and im- plementing delay compensation strategies to minimize the sim2real gap. We validate our approach
real-world systems, demonstrating its effectiveness in improving sim2real performance by accurately modeling both system dynamics and delays. Our results show that the proposed framework supports both accelcrated simulation and real-time processing, making it valuable for robot. learning.
1 Introduction
Sim2real, the transfer of control policies from simulation to the real world, is crucial in robotics thanks to its ability to solve tasks efficiently without the risks associated with real-world learning (Tan et al., 2018; Rudin et al., 2022). With recent advancements in physics simulation on accolerator hardware (Todorov et al., 2012; NVIDIA, 2020; Hu et al., 2020; Freeman el al., 2021), parallelized simulations have greatly reduced training times for complex tasks (Rudin et al., 2022; Hoeller et al., 2024). However, discrepancies between simulation and reality, such as unmodeled dynamics, often reduce the effectiveness of these policies in real-world applications. Addressing this 'sim2real' gap is essential for effective transfer of policies from simulation to the real world.

---

Published in Transactions on Machine Learning Research (2/2025)
Agent
Agent
Agent
8
Eralttnn
Estimator
Agent
Simulator
Real World
Simulator
Real World
Vele
C-eel t-
tarlt
(a) Naive
(b) Real world
(c) Delay simulation
(d) Delay compensation
Figure 1: A policy trained in simulation (a) may perform suboptimally in the real world (b) due to delays. The notation uetralt-r, denoles that an action u is applied at L ++ Ta based on information up L - Ta, where Tn and To are the actuation and sensing delays, respectively. By simulating these delays during training (c), the sim2real gap with (b) can be reduced. A Alternatively, an estimator (d) can predict future states and compensate for delays, improving policy transfer from (a) to (d). The notation Frtrult-r, denotes that state is predicted at & Ta based on information up to to Ta.
A critical yet often overlooked issue sim2real transfer is the impact of lateney in real-world systems, which can degrade performance (Tan et al., 2018; Ibarz et al., 2021; Elocla al., 2023; van der Heijden et al., 2024b). The real world is inherently asynchronous, with delayed sensor data causing agents to act on outdated information. Additionally, slow policy evaluations can further delay the agent's 's actions, compounding these latency issues and leading to suboptimal performance. To mitigate these effects, Fig. 1 illustrates two common compensation stralegies: simulating delays during training (Fig. lc) (Tan et al., 2018; Elocla et al., 2023; van der Heijden et al., 2024a) and using an estimator to predict future states (Fig. 1d) (Smith, 1957; Sherback et al., 2006; Augugliaro et al., 2014; Yang et al., 2020). However, both strategies have limitations. Delay simulation complicates training because the agent's input must include a history of observations and actions restore the Markov property, while an estimator requires accurately modeled system dynamics and delays, which are often difficult to identify (Unbehauen & Rao, 1990).
The hierarchical and asynchronous nature of robotic systems further complicates accurate and efficient sim- ulation on accelerator hardware. Unlike conventional RI, which assumes a single, synchronized environment (Brockman et al., 2016), robotic systems consist of interconnected models operating at different rates, with asynchronous communication introducing complexitics like inter-model latencies and stochastic dynamics (Quigley al., 2009; Baheti & Gill, 2011), leading to irregular computation patterns. Irregular execution paths require scrialization, reducing GPU efficiency, and while simulating time-scale differences improves sim2real accuracy, it further exacerbates this inefficiency (Shibata, 2010).
The main contribution of this paper is a sim2real framework, REX (Robotic Environments with jaX), that introduces& a graph-based simulation model with latency effects, optimized for parallclization on accelcrator hardware. The Iramework's 's innovation lies in its ability to simulate asynchronous, hierarchical systems by explicitly modeling computation, communication, actuation, and sensing delays, while incorporating delay compensation strategies for improved sim2real transfer. Parallelization in both state and parameters allows for simultancous estimation of system dynamics and delays from real-world data, efficiently minimizing the sim2real gap. Additionally, it supports real-world deployment by distributing computations across CPU cores and accelerators, optimizing for latency and performance.
For RL and robotics practitioners, this framework offers several advantages. It enables the modeling of both simulated and real-world systems through a unified, ROS-like graph-based pipeline (Quigley et al., 2009). The framework supports accolerated training speeds familiar to RL workflows and reduces the sim2rcal gap by refining models with real-world data. Integration with the JAX (Frostig et al., 2018) ecosystem further supports advanced RL training and optimization (Lange, 2022b;a; Tang et al., 2022; Lu et al., 2022).
Building on these advantages, we make four key claims: Our framework (i) enables the identification of both dynamics and delays from real-world data, (ii) implements delay compensation and simulation techniques that are essential for effective sim2real transfer, (iii) facilitates efficient parallelized offline simulation on accelerator hardware, (iv) supports real-time online processing capabilitics that meet the latency and per- formance requirements of real-world systems. These claims are supported by experiments on two real-world systems. The pendulum swing-up task clearly demonstrates how neglecting delay simulation can impair
2

---

Published in Transactions on Machine Learning Research (2/2025)
policy transfer, highlighting the need for delay-aware approaches, while the quadrotor task shows scala- bility to more complex robotic systems. The documentation, tutorials, and our open-source code can be found at https://bheiidee atthub.iorresi A video recording of the real-world experiments is available at ftttp:://youuu.be/7]]53
2 Related Work
Sim2Real Frameworks Sim2real frameworks such as Orbit (Mittal et al., 2023), Drake (Tedrake & the Drake Development Team, 2019), and EAGERx (van der Heijden et al., 2024b) facilitate the transfer of control policies from simulation to real-world settings. However, they generally do not include direct support for delay or dynamics identification from real-world data. Our framework addresses this gap by integrating these capabilities directly into the framework. Orbit utilizes Nvidia PhysX for parallelized simulations on accelerator hardware (NVIDIA, 2020). Our framework is based on JAX (Frostig et al., 2018) to support parallelized computation accelerator hardware, while also enabling automatic differentiation. Moreover, our framework, like EAGERX (van der Heijden et al., 2024b), is not restricted to a specific simulator, long as the simulator is compatible with JAX, such as Brax (Freeman et al., 2021) or the MJX extension of MuJoCo (Todorov et al., 2012). This flexibility enables users to select and extend engines as needed within the graph-based model. Tab. provides a feature comparison between REX and related sim2real frameworks.
Delay Estimation System identification involves estimating the system's 's dynamics from input-output data and is a well- established area research (Ljung, 1998). Traditional methods primarily focus on linear systems, often utilizing least-squares optimization techniques (Van Overschee & De Moor, 2012). while more recent efforts have extended to nonlinear systems (Nelles, 2020). Recent advances leverage the differentiability of general-purpose simulators to estimate complex system dynam- ics (Le Lidec et al., 2021; Heiden et al., 2022; Caluwaerts et al., 2023). Our approach builds on these ad advancements by exlending simulators with delay dynamics, allowing for the joint estimation of both system dynamics and delays. Instead of gradient- based methods, we use evolutionary strategics (Hanson, 2006), which we found to be less susceptible local minima and better utilize the parallelism of modern hardware (Tang al., 2022).
REX Orbit Drake LAGERx
Multi-Sim Compatible
GPU Accelerated
/
Gradient Information
V
x
/
Delay Simulation
/
/
Delay Estination
X
Dynsmics Estimatton
X
X
Table A feature comparison between REX and related sim2real frameworks.
Delay Simulation Frameworks like Drake, EAGERX, and Orbit provide support for fixed delay simula- tion (Tedrake & the Drake Development Team, 2019; Mittal et al., 2023; van der Heijden et al., 2024b). Our framework, however, extends this capability by supporting stochastic delay simulation using Gaussian Mix- ture Models (GMMs). Additionally, incorporates correlations between delays by considering the system's 's topology and communication structure during simulation. Although our framework allows for correlated delays, these delays are data-independent and do not change based the simulated data. For example, even if an object delection algorithun takes longer to process when multiple objects are in view, our simulated delays remain the same regardless of the number of detected objects.
Delay Compensation Delay compensation in sim2real has been addressed through various methods. Algorithmic approaches for compensating delays have been proposed by Schuitema al. (2010); Bouteiller et al. (2021). Other studies have enhanced sim2real performance by simulating delays during training and using a history of observations and actions as policy inputs (Tan et al., 2018; van der Heijden et al., 2024b). As part of their approach, Rudin et al. (2022) modified IsaacGym (Makoviychuk et al., 2021) to include a custom actuator model from (Hwangbo el al., 2019), which accounts for control signal delays caused by hardware/softtwar layers. These methods teach policies to handle delays without compensating for them directly during real-world execution. Direct compensation techniques, such as the Smith predictor (Smith, 1957), have long been used in roboties to manage delays from sensors, actuators (Sherback et al., 2006; Augugliaro et al., 2014), and planning latency (Yang et al., 2020). In our work, we demonstrate that by
3

---

Published in Transactions on Machine Learning Research (2/2025)
computation delay
M
communication delay
Estimator M Saley Irre
Agent
Motor M caley Frsd
M
M
M
M
M
deta ore
Skley mnd
Camera 60 Hz
Estimator
Agent Hz
Motor 50 Hz
sensor deloy
A
simulation stepsize
actuator delay
dRiy Jree
(a) Real-world system
(b) Simulated system with delays
Figure 2: Comparison between real-world system setup and simulated system with integrated delays. The real-world system (a) operates with different nodes at specified rales, while the simulated system (b) incorporates various types of delays to closely mimic real-world timing behaviors, including sensor, actuator, commuiiattion, and computation delays.
compensating for delays during execution, can eliminate the need for delay simulation during training, resulting in more efficient training process while maintaining high performance in real-world applications.
3 Our Sim2Real Framework
In this section, we present our framework for sim1reel transfer in robolics, focusing on accurately modeling and compensating for the asynchronous interactions and delays encountered in real-world systems. In the following, we will first describe the graph-based architecture that facilitates asynchronous message passing and delay modeling. We will then detail the three runtime configurations designed for simulation, accelerated training, and real-time deployment. Finally, we will cover the integration of system identification techniques and delay compensation strategies to bridge the gap between simulation and reality.
3.1 Overview
The central element of our framework is the node, which represents a discrete wnit of computation or sensing, operating asynchronously within the system. Nodes are designed run at specified rates, processing inputs, maintaining state, and generating outputs. In approach, both real-world and simulated systems are implemented as networks of these nodes, where communication occurs via directed edges, shown in Fig. 2a. Each node's operation is defined by step function that determines its behavior, transforming inpuls into outputs. For example, nodes can represent various components such as cameras, agents, or motors, each handling spocific tasks like sensing, control, or actuation. This modular design allows for flexible state, time, and action abstractions, supporting the modeling complex interactions in a recentralized manner. Nodes are interconnected in directed graph, facilitating asynchronous message passing and enabling nodes operate at different rates. This design also enables the swapping of real-world nodes with simulated nodes, resulting in unified software pipeline that can be used for sim2real transfer.
Asynchronous operations are inherent in real-world systems due to network transmission times, processing lags, or mechanical response times, which introduces delays into the system dynamics. To address this, we introduce a delay simulation model that captures both deterministic and stochastic delays, incorporating realistic timing behavior through delay distributions for communication, computation, sensor, and actuator delays. As shown in Fig. 2b, our model explicitly defines these delays as non-ncgative distributions, ensuring that the liming characteristics of the simulated environment closely match those of the real world. While this provides the structure for delay simulation, the challenge of estimating the correct delay parameters is addressed later in Sec. 3.3.
The framework supports multiple communication prolocols to manage the flow of messages between nodes, allowing users to specify whether each communication channel should be blocking or non-blocking. blocking channel ensures that a receiver node waits for the most recent message before processing, which minimizes latency in real-time systems by avoiding outdated information. However, blocking channels can introduce instability if delays cause unforeseen propagation through the graph; in such cases, non-blocking channels may
4

---

Published in Transactions on Machine Learning Research (2/2025)
clasu Aganz (Bwseldde);
Baul roarld undes
# Bisaa ato podea consect Lons
def init (rlln, mag graph_ stabe): return PyIree(a".... br..:)
con Camora (rate-60)
can
Samdax (rate-80, delay- (o. 05, 0.0133
agent Agent (rate-30)
agent Agent (rata=31,, delay- 02, 0.0333
notar
Motar (rate:50)
antor Simlotor (raba-50, Gelny-Gamaa 04, 0.013)
caat,, 22g. srapt_ctate); return PyI ooe 2
aodes
[can, agent, aotoz]
brax Braxirate 100, deiay Dssermas1scic(00000
Connect
nodee [bras, can. sgent, potoz] 0.013)
agent, (cam)
toging
delay Gaaaaa-Guuaaaa. cocamet aulay-Gauaa
pnit.putput (aelf, 1ag, staph_
Last maga
0.01))
Fytrou(y1
block
window 2)
m05or (cane, delay Gauss (0.01, 01, 0.01)
10
Funtire: WALL CLDX
10
delay-GGuue (0.01, 0.01>,
11
km jit jitncongilla le with v
11
Uledd for raal rral-tiie opareti
11
windme-9, block-True)
12
def ctop ctop (top_atate)
12
graph Graph(agett, nodec,
12 F Buntine: EIMULATED (no taroccling)
13
cc atop_ atop_etate
enorton nape
13
Clock VALL CLOGS,
13 graph Graph (agez,, nodes, Clock SIMILATED,
14
* Read parans, and current state paraza, atate an parans, su state
10
RealTtnefactor REAL TIME)
14
Resliimeeaçtor FAST, AS PO8S
15
15
Aloud-of- coopl lacion of
15 graph varzup (dev $ 3IT compilasion
16
Ourrent opisode, Seguncee Linesrapp
16
I every Rode lo a atte() nethod
16 go graph init() Graph stace
17
apx, neg. ix en epx, Kn. neg, Xx. ta
17 graph varrop cak
17 for
range
Simmlates 100 staps
Grah the date, and
hinwatanga
18
Fun the graph agentla rate
18
gx graph ran(ga)
19
can inguta can') uonnectd unde
13
g4 graph inis() Graph atate
10 graph stop()
sodea
can data, can sene, Cax Te,T
20 for range
20 birmisted
conpiration graph
21
Compune prate and outpot nep
ge graph (ge)
CF graph get record graph() angaent (rodes)
graph stop()
paraliel graph CoupiledGraph (agest,
23
cutput Pytrou(y1
y2
foutpate, Hielngo)
23
Update step_ for stop call
20 rocord graph get_rocard()
24
split(
PRNCkey nuz-1000)
25
raplace (stata atate)
25 Conwart
Flow
26
weap ini (rugs) 5 todn
26
return auzput Sandla outgu:
26 ac record graghh)))))
26
rullout Snapggrpph rollant) (gaa, enga) run
(a) Node definition
(b) Real-world runtime
(c) Simulation runtimes
Figure 3: Node definitions (a) use generic PyTrees that allow for compilation across different architectures for reduced latency. Examples runtime configurations, showing different execution modes: WALL _ALL_CLOCK
CLOCK (b, Line 13) for real-time operation on physical hardware, SIMULATED (c, Line 13) for simulating without real-time constraints, and COMPILED (b, Line 23) for parallelized execution on accelerator hardware. Variable names and notations were slightly shortened for clarity and space.
be preferable. For example, an estimator node might opt for non-blocking behavior to continue predicting the system's 's state when sensor messages are delayed, allowing the controller to maintain responsive operation.
3.2 Runtimes
Our framework leverages JAX (Frostig et al., 2018) for efficient computation, utilizing its ability to perform just-in-time (JIT) compilation and automatic differentiation, which are crucial for high-porformance machine learning applications. Nodes are defined using generic interface, with parameters, states, and outputs specified using data structures that can be statically analyzed, as shown in Fig. 3a. This approach allows for ahead-of-time (AOT) compilation of the step method (Fig. 3a, Line 12) on various architectures, including CPUs and GPUs, thereby reducing latency. By compiling nodes in this manner, they can be seamlessly employed across different runtime modes without modification, ensuring flexibility and efficiency in both roal-world and simulated environments. Our framework supports three distinct runtime modes, each tailored for different stages of development, training, and deployment: CLOCK, SIMULATED, and COMPILED.
The WALL_ WALL_CLOCK CLOCK runtime is designed for real-time execution on physical hardware, operating at real-time speed with each node's step function rumning asynchronously at its designated rate (Fig. 3b, lines 1-14). Nodes can be compiled to run dedicated hardware resources such as separate CPU cores or accelerator hardware, minimizing latency (Fig. 3b, Line 17). After initializing the state of the graph, which aggregates the states of all nodes, the graph can be executed for a specified number of steps while recording the outputs and their corresponding timestamps (Fig. 3b, lines 19-24).
The SIMULATED runtime enables thatr--thn-- simulation, allowing for accelcrated testing and devel- opment without real-time constraints. Message passing is based on simulated timestamps that are generated based on the communication protocol of every connection (blocking non-blocking) and specified delay distributions, replicating real-world asynchronous effects (Fig. 3c, lines 1-14).
The COMPILED runtime further leverages accelerator hardware like GPUs or TPUs for parallelized execution by enabling the compilation of entire computation graphs into a single function. This makes this runtime suitable for tasks such as training RL policies and largo-scale system identification that can leverage massive parallelism. Data flows from other runtimes (e.g., (Fig. 3b, Line 26)) are converted into a computation graph (Fig. 3c, Line 21) ) and compiled for parallel execution (Fig. 3c, lines 22-26), encoding the asynchronous effects
5

---

Published in Transactions on Machine Learning Research (2/2025)
And "trrr-Comere" according to trainchle nin/mox deloy specifitation
ar argmin
caloulate lces L
Actions
Camera
B, 8,4,0
Agent
Agent
Estimotor
Estimator
Time
Time
Camera
Camera
Agent
Estimator
Brax
Brax
Motor
Camera
Motor
Messs St.
"Moter-a Brec" edges
Time (6)
Time (s)
ascorting to assumed deley
octions
Time (s)
(a) Collected data and data flow
(b) Computation graph
(c) Optimize
Figure 4: System identification example applied to the system in Fig. 2a. (a) Data collection from the real-world system, including sensor data and timing information. (b) Construction of a computation graph that integrates the data flow with simulated nodes for dynamics and hidden delay ifeeatii.ation. Motor-Brax edges are added based on specified delay distribution, while Brax-camera edges follow a trainable min-max delay specification. (c) Optimization of simulation parameters and delays to minimize discrepancies between simulated and real-world behaviors, locusing on the delay interpolation parameter a and the parameters 1 9 for the camera and Brax nodes.
of real-world interaction or simulated delays and enabling parallel execution on accelerator hardware. By supporting these three runtime modes, our framework provides comprehensive flexibility for wide range of applications, from real-time deployment parallelized system identification and policy training.
3.3 System Identification
System identification is crucial for minimizing the sim2real gap by ensuring that the simulated model closely mirrors the real-world system. Our framework facilitates this by identifying both the dynamics and delays inherent in real-world systems, allowing for more accurate simulation and effective delay compensation. In the following, we detail how to build and optimize a tailored computation graph from the real-world data collected to estimate system dynamics and delays (see Fig. 4). Data is collected from the real-world system using the WALL_CLOCK runtime, logging not only sensor and actuator data but also the timing information associated with message exchanges between nodes. This includes the timestamps for when a message is received, when a node begins processing, and when sends out the output. Using this data, we construct a data flow graph that captures node interactions, including the precise timing of messages (sce Fig. 4a, and Line 21 in Fig. 3c).
Dynamics The data flow graph serves as a foundation for identifying the system's 's dynamics. One ad- vantage of using a data flow graph that it inherently represents asynchronous interactions and correctly encodes time-scale differences between nodes. Accounting for these asynchronous effects is essential, as they significantly impact the identified system dynamics (Unbehauen & Rao, 1990). Given the data flow graph, our framework builds a tailored computation graph as follows. We augment the data flow graph with a simulator that models the system dynamics by adding simulator nodes at the desired simulation rate. Edges between simulation nodes and weallwordd-nnleracting nodes are introduced to pass the simulation state to the nodes that model real-world interactions (actuators, sensors, etc.), according to the assumed delay distributions, as shown in Fig. 4b. These delay distributions are either trainable or prespecified, as explained later in this section. By replaying actions through the computation graph and comparing the reconstructed outputs with the collected data, we optimize the simulator parameters to minimize reconstruction loss. During this process, all parameters within the computation graph (e.g. simulator parameters those in any other nodes), can be optimized. For instance, in the example shown in Fig. 4c, we simultancously idcntify Brax's system parameters and the camera's parameters for angle-to-pixel conversion, but we could have also optimized for any other parameter in the graph, such as the motor's friction. The COMPILED runtime is par- ticularly advantagoous for this optimization process due to its ability parallelize computations efficiently. We found ovolutionary strategies effective for this task, as they leverage parallelism, constraint specification, and are less susceptible to local minima (Hansen, 2006; Tang et al., 2022; Lange, 2022a).
6

---

Published in Transactions on Machine Learning Research (2/2025)
Measurable Delays In addition to dynamics, our framework addresses delay estimation, distinguishing between directly measurable delays and hidden delays, such as those in actuators and sensors. Using the recorded timing data, we estimate the communication and computation delays of the system by fitting a Gaussian Mixture Model (GMM) to the measurable delay data using gradient descent. Details on the GMM fitting can be found in Appendix A. Typically, around a thousand samples are sulficient for fitting, which, depending on the system's 's rate, may require less than a minute of data collection. When sampling from the GMM, we clip the sampled values to be non-negative, as delays are inherently non-negative.
Hidden Delays With hidden delays we mcan delays that are not dircctly obscrvable in the data flow graph, such as delays belween the real world and sensors actuators. While we support the addition of edges between simulator and weal-wwold-interacceee motor-Brax connections in Fig. 4b), users can also introduce trainable delays to identify hidden delays (e.g., Brax-camera connections in Fig. 4b). Our approach requires specifying minimum and maximum bound for each trainable delay, which we use to introduce additional edges that accommodate all possible communication patterns between two nodes under minimum and maximum delay conditions. We then introduce a trainable parameter E [0, for each connection, allowing interpolation between the minimum and maximum scenarios. Different deterministie interpolation schemes, such linear zero-order hold, are currently supported to model various delay characteristics.
nodes based on prespecified delay distributions (e.g..
3.4 Delay Compensation
Once the system dynamics and delays are identified, the framework supports various strategics for delay compensation to enhance sim2real performance.
Delay Simulation One straightforward strategy is to integrate the identified delay distributions into the simulation environment. This approach, referred to as delay simulation (Fig. 1c), allows the agent to learn policies that are delay aware. Notice that delays make the problem non-Markovian. To address this, a history of observations and actions can be stacked and used as input to the policy to restore the Markov property. This does make the learning problem more challenging, as the agent must learn to solve the task and handle delays simultaneously, as will show in experiments.
Estimator While RL approaches often treat the environment as a black box, in sim2real scenarios, utilize the identified system dynamics and delays to design a model-based delay compensator that predicts the system's behavior during real-world execution. Inspired by a Smith Predictor (Smith, 1957) and shown in Fig. 1d, our strategy is to predict the state we expect when the corresponding command based on this state reaches the system. By knowing all delays, we can predict when a command will arrive and estimate the system state that future time. Specifically, when a sensor captures an observation, we timestamp it and subtract the identified hidden delay T, to estimate the timestamp of the world's state the observation corresponds to, t.- T. When the estimator processes the observation at fe, it can determine when the resulting command will reach the system by adding the expected estimator-to-actuuuee in te + Ta. Thus, the estimator first updates the state up to t, Ta and then predicts it forward to te + using the past control inputs and their estimated timestamps. We recommend using an Unscented Kalman Filter (UKF) (Julier & Uhlmann, 2004) for this task bocause effectively handles non-diffcrentiable and non-linear dynamics, while requiring only a small number of particles that can be efficiently evaluated in parallel (see Appendix B for more information). Additionally, in partially observable settings, a UKF can infer the hidden state of the system from observations and provide this state the agent, cnabling training in fully observable, delay-free environment, which generally facilitates casier learning. In our experiments, we will evaluate the benefits of using such an estimator for delay compensation and compare the performance gains of delay compensation alone versus delay compensation with hidden state estimation.
latency, Tap resulting
3.5 Limitations
Our framework does not support running nodes on different machines; computations are restricted dif- ferent. devices via JAX. This limits the ability to compile nodes for low-level controllers onboard robot.
7

---

Published in Transactions on Machine Learning Research (2/2025)
Additionally, JAX's Just-In-Time (JIT) compilation can lead to long compilation times, although recent updates with function caching have mitigated this to some extent.
The framework estimates hidden delays as deterministic, which is a reasonable assumption for many robotics applications. Nevertheless, stochastic delays can be modeled by adding variability to the deterministically identified delays, for example, to simulate jitter in sensor readings. Also, our approach requires setting minimum and maximum bounds for trainable delays, but we have found that using large bounds often yields good results. Furthermore, our delay simulation is state independent, meaning that while it accounts for the correlation and stochastic nature of delays, it does not adapt to the specific conditions or data of each simulation step. For instance, if an algorithm takes longer to process when there are multiple simulated objects in view, our approach would not capture this increase in processing time that would occur in a real-world scenario.
As systems scale to large configurations, efficiently parallclizing full asynchronicity for every node can become challenge. To this end, we leverage the supergraph approach in (van der Heijden et al., 2024a) to efficiently parallelize graph-based based simulations. Furthermore, the graph-based framework provides flexibility by allowing users to adjust the level of detail as needed for the task. For example, users may model entire robots as single nodes, focusing on interactions between them rather than internal asynchronicity, to maintain scalability in large-scale systems. The complexity of calculations within nodes is efficiently managed using JAX (Frostig et al., 2018), which enables scalable computations through its support for parallelization and distributed computing across multiple devices.
4 Experimental Evaluation
The main focus of this work sim2real framework that addresses asynchronous interactions in real-world systems by modeling dclays and using real-world data for accurate system identification and reinforcement learning training. Our experiments are designed to validate the key claims made in Sec. 1 as follows. First, we identify the system dynamics and delays from real-world data, followed by a sim2real transfer evaluation using the identified system while using delay compensation techniques. We validate our approach on two distinct real-world systems: a pendulum swing-up up and a quadrotor control task.
4.1 System Identification and Delay Estimation
To support the claim that our approach enables the identification of both dynamics and delays from real- world data, we present system identification and delay estimation results for the two selected systems.
Pendulum In contrast to the classic swing-up task (Brockman et al., 2016), which uses full state infor- mation, our setup relies solely on camera images of the pendulum. This lask highlights the challenge of delay estimation and system identification from images. We apply an open-loop voltage sequence to the motor for 21 seconds while recording stream of images from RealSense d435i camera, in addition to the applied actions and corresponding timing information. Using this data, we construct a data flow graph that is augmented to form a computation graph, incorporating simulator nodes operating at 100 Hz. We introduce edges between the simulator nodes and the camera and motor via two trainable delays that as- sume a minimum and maximum delay of 0 to 50 ms, respectively. Images are first preprocessed through background subtraction and color thresholding to detect the center pixel coordinates of the red dot that marks the pendulum's mass.
The actions are then reapplied to the simulator, and we optimize the parameters to minimize the reconstruc- tion error between predicted and actual pixel coordinates. Simultaneous optimization is performed on several parameters: the physics paramcters of the Brax simulator (mass, length, friction, inertia, etc.); parameters for hidden camera and motor delays, and the parameters of an ellipse model (center, axes, rolation) that maps pixel coordinates to angles using the intuition that the pendulum's motion (as pixel coordinates) will be an cllipse when projected onto the camera plane. A UKF is employed for full state estimation and delay compensation, utilizing a lightweight dynamics model (see Appendix C). We use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Hansen, 2006) to optimize the 27 parameters by minimizing the
8

---

Published in Transactions on Machine Learning Research (2/2025)
encoder
filtered
prodictive
9.0
7.5
Probobality Senatty scall)
(rad)
AM
W
Camera
Estimator
Agent
-7.5
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
19.0
19.2
Time (6)
Time (s)
20
dalay (ma)
delay (ms))
delay (ma)
(rad/s)
(rad/s)
0 M
Brax
Motor
20
25
5.0
7.5
10.0
12.5
15.0
17.5
20.0
19.0 Time (s)
10.2
Determi-iatis deizy
22
Time (#)
dalay (ms)
(a) Reconstruction
(b) Zoomed
(c) Estimated delays
Figure Pendulum system identification and delay estimation. (a) Open-loop reconstruction of the angle (0) and angular velocity (f) with the braz simulator, compared to ground-truth encoder data. (b) A zoomed view shows that the predictive UKF estimate that compensates for delays, outperforms the filtered estimate that does not. (c) Estimated GMM delay distributions and deterministic hidden delays for the camera and motor, with the grey area indicating the measured delay distribution and the black line showing the GMM fit.
MSE
MSE₁
filtered
0.069
4.256
predictive 0.012
1.114
(a) Real images
(b) Rendered images with Brax
Table Mean squared error (MSE) with respect to ground- truth encoder data. Boldface in- dicates the best performance.
Figure 6: Comparison of real and rendered images of the pendulum from two different viewpoints. (a) shows actual images captured from side and frontal views.
(b) shows the corresponding rendered images from the estimated poses.
reconstruction error between the predicted and measured pixel coordinates. See Appendix E for details on CMA-ES and the hyperparameter scttings. Finally, we fit GMMs to estimate delay distributions for all measurable communication ion and computation delays.
The reconstructed angle and angular velocity from the simulator and estimator are shown in Fig. 5a, alongside the validation data obtained from the pendulum's encoder, which can be considered the ground truth with minimal delay. The open-loop reconstruction remains accurate over 21-second time horizon. The identified delay distributions are illustrated in Fig. 5c, with a motor-to-Ban Brax delay of approximately 7 ms and a Brax- to-camera delay of around ms. The camera delay exhibits multi-modal distribution, suggesting variability due to internal processing and shutter speed. The effectiveness of delay compensation demonstrated in Fig. 5b by comparing the filtered and predictive estimates. The filtered estimate shows the UKF's state estimate plotted against the timestamp of when the action using the estimated state was applied to the simulator, resulting in noticeable phase shift of around 50 ms. In contrast, the predictive estimate forecasts the filtered estimate forward, resulting in lower mean squared error (MSE) for both the angle and angular velocity, as shown in Tab. 2. Finally, we use the identified system to render images from the estimated poses, as shown in Fig. 6. The comparison between real and rendered images from two different viewpoints qualitatively demonstrates the accuracy of the estimated system parameters.
Quadrotor Next, we identify the dynamics and delays of a quadrotor system using real-world data to demonstrate the applicability of our approach to higher-dimensional statc-action spaces. The quadrotor's yaw is fixed, while the reference roll and pitch angles and the height setpoint are sent to a PID controller to maintain a circular flight path at a constant altitude. The PID controller converts the height setpoint to a thrust command, which, along with the roll and pitch commands, sent to the Crazyflic. We record the actions, timing data, and state information captured by motion capture (MoCap) system. Similar to the pendulum experiment, construct data flow graph that is augmented form computation

---

Published in Transactions on Machine Learning Research (2/2025)
mocap
mocap (recon)
0.5
0.5
MMM
0.24
(rad)
(rad)
0.20
MoCap
Agent
PID
0.16
-0.50
-0.50
Time (s)
Time (s)
Time (s)
delay (ms)
delay (ms)
WW
Areaity (lay swile)
delay
M
v, (m/ss)
vy (m/s)
(m/s)
ODE
-1.90
delay
15
18 delay (ms)
6.2 Time (s)
6.3
Time (s)
Time (s)
(a) Recomstruction
(b) Zoomed
(c)
Estimated delays
Figure 7: System identification and delay estimation for quadrotor control. (a) Open-loop reconstruction of roll, pitch, and velocilies in body frame over 15 seconds (recon), showing the accuracy of the identified model compared to the MoCap data (mocap). (b) Zoomed view illustrates an accurate fit. (c) Estimated GMM delay distributions, with the grey area indicating the measured delay distribution and the black line showing the GMM fit.
graph with simulator nodes operating at 100 Hz. Edges are introduced between the simulator nodes and the MoCap and PID nodes, incorporating hidden delay nodes with a minimum and maximum delay of 0 to 50 ms, respectively. A dynamics model similar to that used in (Kooi & Babuska, 2021) is employed (see Appendix D). Simultaneous optimization performed on the dynamics parameters (e.g., mass, drag, gain, time constants) and interpolation parameters for hidden delays between the dynamics model, MoCap, and PID controller, using CMA-ES (Hansen, 2006) to optimize the eight parameters by minimizing the reconstruction error between the predicted and measured quadrotor attitude and body frame velocities. See Appendix E for details on CMA-ES and the hyperparameter scttings. We also fit GMMs to estimate delay distributions for all measurable communication and computation delays.
The results in Fig. 7a show accurate reconstruction of the quadrotor's states over 15 seconds. The identified delays are shown Fig. 7c, with PID-to-ODE delay at 34 ms, ODE-to-PID delay 9 ms, and ODE-to- MoCap delay at 15 ms. In the next section, evaluate the advantage of delay-aware system identification for sim2real transfer by training policies with and without considering delays.
4.2 Sim2Real Transfer
To support the claim that our approach implements delay compensation techniques essential for effective sim2real transfer, we evaluate the sim2real performance of policies trained with and without delay compen- sation for the pendulum and quadrotor systems.
Pendulum Swing-Up Up This task highlights the challenge of delay compensation and partial observability in reinforcement. learning. By demonstrating that neglecting delay simulation can impair policy transfer even in scemingly simple scenario, we underscore the necossity of delay-aware approaches for more complex systems, where delays are inevitable and system dynamics are more intricate (Liu al., 2019; Asaamoning et al., 2021; Lou et al., 2019; Peters et al., 2014). The pendulum task's simplicity effectively clarifies the importance of addressing delays in sim2real frameworks.
To investigate the impact of delays and partial observability on task complexity, we train pendulum swing- up policies using PPO (Schulman et al., 2017) under different conditions in simulation (see Appendix F for more details). We evaluate policies trained with full state information, stacked obscrvations with and without delay simulation, and estimated full state information with simulated delays. As shown in Fig. 8a, policies with full state information achieve higher rewards and converge faster than those relying on stacked
10

---

Published in Transactions on Machine Learning Research (2/2025)
delay sim.
no-delay sim.
1
estimator delay
stack
stack delay sim.
full
c
(m)
(ml
o
-102
1.0
1.0
cum. reward
o (m)
x (m)
(m)
success
success
0.5
0.5
(m)
(m)
-10*
0
0 (m)
0
1 steps
sim environment
real
0.0
enc. filt. pred.
(m)
(m)
1e6
observation
(a) = 1.0m (b) 0.75m
(c) R 0.5m
(a) Training curve
(b) Sim2real
(c) Estimator
Figure 9: Sim2real performance of quadrotor policies trained with and without delay sim- ulation across different path radii. Top row: simulated path following at radii R = 1.0m,
Figure Sim2real evaluation of policies trained under different de- lay and observation conditions for the pendulum swing-up task. (a) Training curves comparing policies with full state information and stacked observations. (b) Sim1real performance showing the per- ventage of time the pendulum remains upright (within 1 and +0.5 rad/s). (c) Performance of a policy using full state estima- tion with delay compensation, demonstrating the importance of delay compensation for steady performance.
R 0.75m, and 0.5m. Boltom row: real- world path following shows that delay simula- tion improves performance and stability, par- ticularly at smaller radii where delays signifi- cantly impact control.
observations, especially when delays are present. This highlights the additional challenge introduced by delays and partial observability, beyond the complexity of the task itself.
Zero-shot evaluations on the real system show that policies trained solely with stacked observations fail to consistently swing up the pendulum, while the policy trained with delay simulation, delay compensation, and full state estimation achieves reliable swing-up, as demonstrated in Fig. 8b. Interestingly, even a policy trained on the full state without simulated delays can achieve consistent swing-up when worldworld evaluation uses an estimator that compensates for delays and estimates the full state, as indicated by pred. in Fig. Sc. We assess the performance gains of delay compensation alone versus delay compensation with hidden state estimation, by evaluating the full state policy in two other scenarios: using the angle encoder, which provides full state information with negligible delay compared to camera images, and using the filtered state estimate from the UKF instead of the sewarrdpredditeed state. Both policies perform suboptimally, suggesting that both full state estimation and forward prediction are essential for reliable performance, as shown in Fig. Sc. A side-by-side comparison of the real-world swing-up performance of the different policies is available in the supplementary video.
Path Following with a Quadrotor We trained quadrotor to fly circular path at maximum speed with varying radii to assess the impact of delay simulation on sim2real performance. We used PPO (Schulman et al., 2017) to train policies using a reward function that penalizes path error and rewards high speeds along the path (see Appendix F for more details). In simulation, all policies achieved successful path following, with the no-delay policy reaching higher speeds and maintaining lower path errors due to its ability to fly more aggressively in the absence of simulated delays, detailed in Tab. 3. However, in real-world tests, only the policy trained with delay simulation maintained stable flight; the no-delay policy exhibited oscillations around the target path. The performance gap widened at smaller radii, with the no-delay policy exhibiting highly unstable flight behavior at a radius of 0.5 m, demonstrating the critical role of delay-aware training for reliable real-world deployment, as shown in Fig. 9. A side-by by-sde comparison of the roal-world flight performance of the two policies is available in the supplementary video.
4.3 Computational Runtime Analysis
To support our claim that the framework enables efficient parallelized simulation on accelerator hardware, evaluated simulation speeds using the COMPILED runtime NVIDIA RTX 3070 Laptop GPU. The
11

---

Published in Transactions on Machine Learning Research (2/2025)
Simulation
Real-world
Radii (m)
Upath (m/s)
Cpath (m)
Ppath (m/s)
Cpath (m)
delay
no-delay
delay
no-delay
delay
no-delay
delay
no-delay
1.00
1.95
2.23
0.03
0.02
2.02
2.05
0.06
0.21
0.75
1.67
1.92
0.03
0.02
1.64
1.63
0.04
0.19
0.50
1.39
1.61
0.04
0.04
1.36
1.18
0.04
0.24
Table Impact of delays on simulated vs. real-world performance across different path radii. Upath denotes the average speed flown along the path, and Epath represents the average error between the quadrotor's position and the target path. Boldface indicates the best performance in each category.
data flow was augmented with simulator nodes and subsequently parallelized to simulate delays according to real-world settings.
We measured the computation time for CMA-ES (Hansen, 2006) converge during system identification for the pendulum and quadrotor tasks. For the pendulum, optimizing 27 parameters with a population size of 200 and 21-second rollout per fitness evaluation (1,050 steps) led to convergence after 38 generations in 22.07 seconds, achicving 380k steps/s with a compilation time of 19.97 seconds. For the quadrotor, optimizing eight. parameters under similar conditions but with a 15-second rollout (375 steps) resulted in convergence after 31 generations in 5.81 seconds, reaching 400k steps/s with a compilation time of 10.16 seconds. We also evaluated PPO training time using the implementation from (Lu et al., 2022): for the pendulum, training five policies in parallel with 64 environments reached 5 million steps in 77.1 seconds (325k steps/s), while for the quadrotor, training with 128 environments for 10 million steps completed in 29.8 seconds (336k steps/s); demonstrating the framework's is efficiency in supporting rapid training on real-world tasks.
To isolate simulation speed from training overhead, we performed parallelized rollout speed analysis (Fig. 10). The results show a linear relationship on a logarithmic scale, indicating that as the number of parallel environments doubles, the simulation speed also roughly doubles. An initial superlinear increase is observed, likely due to constant overheads being amortized over a larger number of parallel environments, resulting in more efficient resource utilization.
The simulation speed in our framework is determined by the computational workload of cach node and the ability to parallelize their interactions. Our framework exlends beyond standard simulations by modeling the asynchronous interaction between components, which are inherently challenging to parallelize efficiently (van der Hoijden et al., 2024a). By demonstrating fast simulation speeds for the pendulum and quadrotor, we show that our framework achieves efficient runtime performance without introducing significant overhead beyond the computations within each node. If the simulation speed were slow, even with the simple dynamics of these systems, would indicate a substantial fixed overhead from the framework.
To compare runtime performance with other sim2real frameworks, we evaluate the runtime of a common system across these frameworks. Specifically, we compare the pendulum system described in Appendix C with the pendulum examples in Drake (Tedrake & the Drake Development Team, 2019) and EAGERx (van der Heijden et al., 2024b), as all systems involve a simple pendulum with a two-dimensional state, ensuring a fair comparison. Since GPU parallelization is not supported in these frameworks, we measure the runtime performance of a single 20-second rollout on CPU while applying random actions. Our results indicate that EAGERx achieves 0.28k steps/s, Drake achieves 60k steps/s, and our framework achieves 15k steps/s a single CPU core. We attribute EAGERx's slower performance to the overhead of ROS communication between nodes, while Drake benefits from its optimized C++ backend. Our framework is significantly faster than EAGERx and, compared to Drake, can support GPU-based parallelized execution, which can further improve simulation speed, as demonstrated in Fig. 10.
To support the claim that our framework meets the latency and performance requirements for real-time online processing in real-world systems, we evaluate the latency of different components during real-world experiments. Our framework records timing information for each node, allowing us to estimate computation
12

---

Published in Transactions on Machine Learning Research (2/2025)
pendulum
quadrotor
Delay (ms)
Rate (Hz)
10a
Node
Device
Computation
speed (staps/s)
Pendulum
106
camera
9.8 = 8.0
60
CPU,
image-lo-angle conv.
estimator
3.5 + 5.2
50
CPU2
UKF update
agent
1.7 = 2.3
50
GPU
Policy NN(64,64)
104
motor
5.6 + 7.0
50
CPUs
Cmd to pendulum
Quadrotor
2
2
24
26
2
2
22
2
parallel envs
mocap
0.3 + 0.1
50
CPU1
Read quadrotor pose
agent pid
2.6 = 0.5
25
GPU
Policy NN((64,64)
Figure 10: Simulation steps per second vs. number of parallel enviromments for 200- step rollout on a GPU. The pendulum (20 ms/step) and quadrotor (25 ms/step) systems both demonstrate a linear scaling in simulation speed with increasing parallel environments.
2.8 + 0.3
50
CPU2
Cmds to quadoopter
Table 4: Delay statistics for Quadrotor and Pendulum nodes, in- cluding delay (mean + std.) in ms, rate Hz, device type, and description of each computation. Subscripts indicate dedicated CPU cores. NN denotes a neural network with layer sizes in parentheses.
and communicatinn delay statistics across both the pendulum and quadrotor systems, as visualized in Fig. 5c and Fig. 7c. The mean and standard deviation of delays for each node's periodic computation are calculated, providing insights into system performance. By dedicating specific CPU cores to each node, we bypass the Python Global Interpreter Lock (GIL), enabling concurrent execution. Additionally, accelerate policy inference in the agent node. This approach results in low latency across the system. Unexpectedly, the motor node in the pendulum system exhibited large delays, likely due to the hardware's slow response time while servicing ROS (Quigley et al., 2009) service calls. As expected, the camera node had the longest delays, attributed to the time required for image retrieval and processing to convert images angles.
use the GPU to
In summary, our evaluation demonstrates that our approach effectively identifies both dynamics and delays from real-world data, compensates for delays to improve sim2real transfer, and facilitates efficient parallelized simulation on accelerator hardware. At the same time, our approach meets the latency and performance requirements for real-time online processing, supporting all four key claims.
5 Conclusion
In this paper, we presented novel framework, REX (Robotic Environments with jaX), for sim2rcal transfer that introduces a graph-based simulation model incorporating latency effects, optimized for parallelization on accelerator hardware. Our approach models asynchronous, hierarchical systems by explicitly representing computation, communication, actuation, and sensing delays. This enables the simultaneous estimation of system dynamics and delays using and evaluated our approach two real-world robotic systems, demonstrating its ability to support rapid training while maintaining high fidelity to real-world conditions. The experiments suggest that our framework not only improves the accuracy of policy transfer by reducing the impact of delays and partial observability but also enhances simulation elficiency by leveraging hardware acceleration.
data, effectively minimizing the sim2real gap. We implemented
For future work, we aim to extend the framework to support estimating stochastic hidden delays, which could further reduce the sim2real gap by more accurately capturing real-world uncortaintics. Additionally, we plan to enhance the framework's scalability and real-world applicability by enabling distributed computing across multiple machines, beyond the current capability of utilizing different devices via JAX.
6 Acknowledgments
This work funded in part by the EU's H2020 OpenDR project (grant No 871449).
13

---

Published in Transactions on Machine Learning Research (2/2025)
References
Godwin Asaamoning, Paulo Mendes, Denis Rosário, and Eduardo Cerqueira. Drone swarms as networked control systems by integration of networking and computing. Sensors, 21(8):2642 2021.
Frederico Augugliaro, Sergei Lupashin, Michael Hamer, Cason Male, Markus Hchn, Mark W Mueller, Jan Se- bastian Willmann, Fabio Gramazio, Matthias Kohler, and Raffaello D' Andraa.. The flight assembled ar- chitecture installation: Cooperative construction with flying machines. IEEE Control Systems Magazine, 34(4))46-64, 2014.
Radhakisan Bahoti and Helen Gill. Cyber-physical systems. The Impact of Control Technology, 12(1): 161 166, 2011.
Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Binas. Reinforcement learning with random delays. In Proc. of the Int. Conf. on Learning Representations (ICLR), 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAT Gym. ariiv preprinl, 2016.
Ken Caluwaerts, Atil Iscen, J Chase Kew, Wenhao Yu, Tingnan Zhang, Daniel Freeman, Suang-Huel Lee, Lisa Lee, Stefano Saliceti, Vincent Zhuang, et al. Barkour: Benchmarking animal-level agility with quadruped robots. arXiv preprint, 2023.
Norhan Mohsen Elocla, Mohamad Chchadch, Igor Boiko, Sean Swei, and Yahya Zweiri. The Role of Time Delay in Sim2real Transfer of Reinforcement Learning for Unmanned Aerial Vehicles. In Proc. of the Int. Conf. on Advanced Robotics (ICAR), pp. 514-58 519. IEEE, 2023.
Julian Förster. System identification of the crazyflie 2.0 nano quadrocopter. B.S. thesis at ETH Zurich, 2015.
C. Daniel Freeman, Frik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-A Differentiable Physics Engine for Large Scale Rigid Body Simulation. In Proc. of the Advances in Newral Information Processing Systems (NIPS), 2021.
Roy Frostig, Matthew James Johnson, and Chris Leary. Compiling machine learning programs via high-level tracing. Systems for Machine Learning, 4(9), 2018.
Nikolaus Hansen. The CMA evolution strategy: a comparing review. Towards a new evolutionory compula- tion: Advances in the estimation of distribution algorithms, pp. 75-102, 2006.
Eric Heiden, Christopher E Denniston, David Millard, Fabio Ramos, and Gaurav S Sukhatme. Probabilistic inference of simulation parameters via parallel differentiable simulation. In Proc. of the IEFE Intl. Conf. Robotics & Automation (ICRA), pp. 3638 3645. IEEE, 2022.
David Hoeller, Nikita Rudin, Dhionis Sako, and Marco Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, :cad)c6)) 2024.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kellcy, and Frédo Durand. DiffTaich:: Differentiable Programming for Physical Simulation. In Proc. of the Int. Conf. on Learning Representations (ICLR), 2020.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms. Journal on Machine Learning Research (JMLR), 2022.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. ICLR Blog Track, 2023.
Jemin Hwangbo, Joonho Lee, Alexcy Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotács, 2019.
14

---

Published in Transactions on Machine Learning Research (2/2025)
Julian Ibarz, Jie Tan, Chelseaa Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. Trill. Journal of Robolics Research (IJRR), 40(4-5):698 721, 2021.
Simon J Julier and Jeffrey K Uhlmann. Unscented filtering and nonlinear estimation. Proceedings of the IEEE, (2(3):401-42) 422, 2004.
Diederik P Kingma. Adam: method for stochastic optimization. ar prepriul, 2014.
Jacob E Kooi and Robert Babuska. Inclined Quadrotor Landing using Deep Reinforcement Learning. Proc. of the TEEE/RSJ Infl. Conf. on Intelligent Robots and Systems (IROS), pp. 2361-2368. IEEE, 2021.
Robert Tjarko Lange. evosax: Jax-based evolution strategics. arXiv preprint, 2022a.
Robert Tjarko Lange. gymmax: A JAX-based reinforcement learning environment library, 2022b. URL
Sttp:///ithut com/RooerttTLang//ggie gymnax.
Quentin Le Lidec, Igor Kalevatykh, Ivan Laptev, Cordelia Schmid, and Justin Carpentier. Differentiable simulation for physical system identification. IEEE Robotics and Automation Letters (RA-L), 6(2) 3420, 2021.
Xing Liu, Hansong Xu, Weixian Liao, and Wei Yu. Reinforcement learning for cyber-physical systems. 2019 IEEE International Conference on Industrial Internet (ICII), pp. 318 327. IEEE, 2019.
Lennart Ljung. System identification. In Signal analysis and prediction, pp. 163 173. Springer, 1998.
Xin Lou, Cuong Tran, David KY Yau, Rui Tan, Hongwei Ng, Tom Zhengjia Fu, and Marianne Winslett. Learning- based time delay attack characterization for cyber-physical systems. In 2019 IEEE Interna- tional Conference on Communications, Control, and Computing Technologies for Smart Grids (Smart- GridComm), pp. 1-6. TEEE, 2019.
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Proc. of the Advances in Neural Information Processing Systems (NIPS), 35:16455-16465 2022.
Viktor Makoviychuk, Tukasz Wawrzyniak, Yunrong Guo, Michelle Tu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning. In Proc. of the Advances in Neural Information Processing Systems (NTPS), 2021.
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments. IEFE Robotics and Automation Letters (RA-L., pp. 1-8, 2023. doi: 10.1109/TRA.2023.22200
Oliver Nelles. Nonlinear dynamic system identification. Springer, 2020.
NVIDIA. NVIDIA PhysX, 2020. URL kurlfatttps://deeelppees www.:..wiiii....... www.iii..co/pyyeeeeeeeeeee
Andrés A Peters, Richard H Middleton, and Oliver Mason. Leader tracking in homogencous vehicle platoons
with broadcast delays. Automatica, 50(1):64-74, 2014.
Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, Andrew Y Ng, et al. ROS: an open-source Robot Operating System. Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), 3:5, 2009.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Proc. of the Conf. Robot Learning (CoRT.), 164:91-100, 2022.
15

---

Published in Transactions on Machine Learning Research (2/2025)
Erik Schuitema, Lucian Busoniu, Robert Babuska, and Pieter Jonker. Control delay in reinforcement learn- ing for real-time dynamic systems: A memoryless approach. In Proc. of the TEEE/RO Inll. Conf. Intelligent Robots and Systems (IROS), pp. 3226 3231. LEEE, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint, 2017.
Michael Sherback, Oliver Purwin, and Raffaello D' A'Anrea.. Real-time motion planning and control in the 2005 cornell robocup system. In Robot Motion and Control: Recent Developments, pp. 245-263. Springer, 2006.
Naoki Shibata. Efficient Evaluation Methods of Elementary Functions Suitable for SIMD Computation Computer science- Reseerchh and development, 25:25-32 32, 2010.
Dan Simon. Oplimal state eslimation: Kalman, H infinily, and nonlinear approaches. John Wiley & Sons, 2006.
Otto JM Smith. Closer control of loops with dead time. Chemical engineering progress, 53:217-214 219, 1957.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint, 2018
Yujin Tang, Yingtao Tian, and David Ha. EvoJAX: Hardware-Acoeleratel Neuroevolution. arXio preprind, 2022.
Russ Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics. attps://drake weiiike..it.eee.. mit.edu, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pp. 5026-5033, 2012.
Heinz Unbehauen and GP Rao. Contimuous-tmme approaches to system identification 26(1):23-335 13:23-3,, 1990.
survey. Automatica,
Bas der Heijden, Laura Ferranti, Jens Kober, and Robert Babuska. Efficient Parallelized Simulation of Cyber-Physical Systems. Trans. Machine Learning Research (TMLR), 2024a. ISSN 2835-8856. URL https://openreviie http://ppenrrree..eeeettttrrrr/t doo.mm/iddaaaXXtccao.. Reproducibility Certification.
Bas der Heijden, Jelle Luijkx, Laura Ferranti, Jens Kober, and Robert Babuska. Engine Agnostic Graph Environments for Robotics (EAGERX): A Graph- Based Framework for Sim2real Robot Learning, IEEE Robolics and Aulomation Magazine (RAM), pp. 2-15, 2024b. doi:
Rudolph Van Der Merwe. Sigma-point Kalman filters for probabilistic inference in dynamic state-space models. Oregon Health & Science University, 2004.
Peter Van Overschee and Bart De Moor. Subspace identification for linear systems: The- ory catpeeettttiio-pppppiiiiiii Springer Science & Business Media, 2012.
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Tingnan Zhang, Jie Tan, and Vikas Sindhwani. Data efficient reinforcement learning for legged robots. Proc. of the Conf. Robot Learning (CoRL), pp. 1-10, 2020.
A Measurable Delay Fitting with Gaussian Mixture Models
Let [a represent the observed delays as one-dimensional data points. We consider a. Gaussian Mix- ture Model (GMM) with components, where each component is characterized by three parameters: (Tes Pk: og). Specifically,
Te: the mixing weight of component k satisfying ΣX Ta and mi 0,
16

---

Published in Transactions on Machine Learning Research (2/2025)
Akk the mean of the Gaussian component,
OR: the standard deviation of the Gaussian component.
The full set of parameters of the GMM is denoted 0 {TT, Pk,, l which includes all mixing weights, means, and standard deviations. The probability density function for single data point 2; under this model is expressed as:
K = N
p(r;
0)
/ [kk o
k=1
where Maz; pgg of) is the probability density function of normal distribution with mean μ4 and variance of To estimate the parameters of the GMM, we minimize the negative log- log-likel of the observed delays;
K
(( =
Σ log Σ
TaN( k=1
Pa: a
The optimization is performed using a gradient- based approach. Specifically, the parameters 0 are updated iteratively using a standard solver such as the Adam optimizer (Kingma, 2014), which adjusts the parameters to minimize (()). To improve numerical stability during optimization, normalize the data before fitting the GMM. The observed delays } are transformed into normalized delays {a{} as:
24 Ho
max e)
where Nr and o₂ are the mean and standard deviation of the observed delays, and is a small constant (e.g., 10-T) to avoid division by zero. The GMM is then fit to the normalized dataset (x() and the negative logelikelihood is computed accordingly. After fitting the model, the parameters are denormalized to map back to the original data range with:
AA P4k Ox + May
ōk o Gz. This approach ensures numerical stability during optimization while providing parameters ta and ōg in the original scale of the observed delays.
B Unscented Kalman Filter
The Unscented Kalman Filter (UKF) is state estimation algorithm designed for nonlinear systems. It employs deterministic sampling technique to approximate the mean and covariance of the state distri- bution. This approach is more accurate than linearization-base methods, such as the Extended Kalman Filter (EKF), for highly nonlinear systems (Simon, 2006). Since the UKF relies on sampling rather than lincarization, can effectively handle non-differcntiable functions. The number sigma points, 2n + 1, is determined by the state dimensionality and is lypically much smaller than the number of particles required for a particle filter. Additionally, the evaluation of sigma points can be efficiently parallelized, The sigma points are calculated to capture the mcan and covariance of the state distribution and follow the procedure in (Van Der Merwe, 2004) to generate 2n + 1 sigma points. Given the current state mean and covariance P, the scaling parameter À = (n points, adjusts scaling, and 33 incorporates prior knowledge (e.g., B = 2 for Gaussian distributions). The sigma points are then determined as:
(n+*k) k)) n is computed, where a controls the spread of the sigma
Xo x,
X = Vn + (P],
1=
where /n X. is the i-th column of the Cholesky decomposition (n+ A)P. The corresponding woights for mean and covariance are:
(m)
A
(c)
(m) + +(1 a² +),, w; = (c)
(m)
Wo = wh
2(n + X) i= 1,...,.. 2n
X
17

---

Published in Transactions on Machine Learning Research (2/2025)
The filter is initialized with initial state mean Xo and covariance Po- The state transition function f()) and observation function k(.) are nonlinear, describing the process dynamics and measurements, respectively. Process and measurement noise are assumed to be additive, Gaussian, and uncorrelated, with zero mean and covariance matrices Q and R. respectively. Actions commanded at time 1 will affect the state at time + Tas where Ta is the actuation delay. Similarly, the most recent measurement available at time is from time t Tx where Tи is the sensor delay. Hence, the indexing of the measurements and actions is shifted by the sensor and actuation delays. For simplicity, we assumne that the sensor delay тa and actuation delay To are constant and multiple of the timestep. Furthermore,
consider time-invariant noise distributions, process models, and measurement models. Note, however, that the UKF can be extended to handle time-varying noise covariances, models, variable delays, and non- non-uniform timesteps, but we do not consider them here for nolational clarity.
The UKF estimation step at time t is outlined Alg. In the prediction and update steps, the UKF updates its prior state estimate from t- Ts to the last mcasurement time, t- Ta: using the most recent measurement available al ( Ts Next, the UKF forward-pred paraardc---ecssss the state to ( + Ta: incorporating the commanded actions available up to t
to account for the combined sensor and actuation delays.
Algorithm 1: UKF Estimation Step with Delay Compensation
Input:
Procoss noise Q mcasurement noise weights (!(")), w process model f(),, measurement model h(-), sensor delay Tu, actuation delay To
(e)
Input: Previous state mean x,-1-r,; covariance Pt-1-ru: new measurement ze-r,, action sequence {ue (us)a---1-g,
Output: Estimated state mean ,,, covariance Pt-z,i forward-predicted state mean Nt4 covariance P4t..
Calculate Sigma Points:
Compute sigma points {x (t-1-1m) from current state mean X;-1-7, and covariance Pt-1-T--
Prediction Step:
Propagate sigma points through process model with actions: x; = 1(x: =[xx (1-1-ra) (1-1-57
Compute predicted mean: xi-r, E, (m))
Compute predicted covariance: P
E, w 10 (x, -x -xx,, - x(-r.) +QQ
Update Step:
Propagate sigma points through measurement model: x, = h(x,)) ).
Compute predicted mcasurement mean: 2,-r, Σ, w/m") (1))
ze.
10 Compute innovation covariance: S = E, w; [z: 2 2, (z: 10
(c)
yT R.
Compute croas-covariance: C Σ, w (0)
(x, -x,-,, )2i -2,r,, y7.
12 Compute Kalman gain: K = CCS
13 Update state mean: Xt-T, x,-T,, + K(a=-r x--r))-
14 Update covariance: Pt-r,- =P KSK?.
15 Forward-prediction Step:
16 for ls = f Ta.....t To do
17
Compute sigma points ( from state mean X and covariance Ps.
18
Propagate sigma points through process model with actions: X; f(x;') .u).
(N)
Compute orward-predictes mean: X4+1 = E. (m()) X;.
19
Compute forward- covariance: P E, w0 O xxx-1(07 - -x++1)
20
21
end
22
return Xt-
PE- 81tra PLtte
C Pendulum Dynamics
The pendulum system is modeled by second-order ordinary differential equation (ODE). The state x = (0,0) represents the angle 0 and angular velocity The control input u represents the applied voltage. The angular acceleration is given by:
K + mgl (i) 1d 0 K² esign(0)
Ø RR
J
where J is the moment of inertia, m the mass, I the pendulum length, b the damping coefficient, K the motor constant, R the motor resistance, the static friction, and g 9.81 m/s" the gravitational acceleration.
18

---

Published in Transactions on Machine Learning Research (2/2025)
Dynamics are simulated using fourth-order Runge-Kutta integration method with fixed time step of 0.01 seconds. All parameters (J, m, I, b, K2 R c) are identified experimentally in this paper except for the gravitational acceleration together with any additional parameters required for hidden delay estimation.
Quadrotor Dynamics
The dynamics model, similar to that in Kooi & Babuska (2021), is used to simulate the quadrotor's motion. The dynamics are divided into three components: rotational, translational, and motor dynamics. The state is represented by the position = (z, y. 2), velocity t (ž, y. 2), attitude = (ø, 0, v)), and thrust state Q. The control inputs are the reference pulse-width modulation (PWM) motor signal and two reference angles &ref and Oret. Yaw dynamicc are neglected, with yaw angle assumed constant at vi = 0. The rotational dynamics are approximated by a first-order system for the attilude angles & and
kelOret 6)
ke 0)
where the same ke and Te used for the rotation in both angle directions due to the system's symmetry, These dynamics comprise the quadrotor's closed-loop onboard control of the attitude angles. The total thrust generated by the quadrotor's motors is modeled as first-order system:
② = am bm
fthrust = C d
where Omn dm, Cm, and dm are motor-specific constants. The drag force acting on the quadrotor given by:
Kayl
0
Sara
0
Kayyl
Ub.
0
O
KeW
where Key and K₂ are drag coefficients, w the rotor speed, and Ub is the body-frame velocity. The body- frame velocity calculated as vs RRvT, where R is the rotation matrix from the body frame the world frame. The rotor speed is approximated using the following relationships (sce Förster (2015) for more details) between the effective PWM signal Oeft, rotor speed w, and total thrust fehrust:
fthrust = (aa, O + 6p 0et cp)
= A(a, + be)
where aps 6p. and cp are PWM constants, a, and b, are rotor constants, and the factor accounts for the quadrotor's four rotors. The translational dynamics are:
R(00, O, ftarust] + fdrag) [0, 0. m
where m is the quadrotor's mass, and g is the gravitational constant. Dynamics are simulated using a fourth-order Runge-Kutta integration method with fixed time step of 0.01 seconds.
The motor, PWM, and rotor constants are specific to the motor and require additional sensors for accurate identification. We use the experimentally identified values from Förster (2015) for the Crazyflie 2.0 quadrotor, as done in Kooi & Babuska (2021):
6m 15.47,
bm 1.0,
Gm 1.43 10
dm 2.89 x 10-7
ap = 2.13 x 10
bp = 1.03 x 10
Cp = 5.49 x 10
a, = 0.041,
6. = 380.83.
The remaining parameters (m, ke, Te: Kays Kg) are experimentally identified in this paper, along with any additional parameters required for hidden delay estimation.
19

---

Published in Transactions on Machine Learning Research (2/2025)
E
Covariance Matrix Adaptation Evolution Strategy
The Covariance matrix adaptation evolution strategy (CMA-ES) algorithm is a stochastic, derivative-free optimization method well-suited for non-linear or non-convex problems (Hansen, 2006). evolves a popu- lation of solutions by sampling from a multivariate normal distribution, adapting the covariance matrix and step size to guide search directions efficiently.
The algorithm steps are outlined in Alg. 2, where the key hyperparameters are defined as follows. First, the total number of iterations generations, denoted G, determines how long the algorithm runs. Each iteration involves evaluating population of candidate solutions, the size of which is specified by x. From this population, the top-performing solutions are selected for recombination, with the condition s X. The learning process also depends on several adaptation rates: Co and dg control the adaptation the step size, while Ce2 C1 and Cn influence the covariance matrix updates, ensuring efficient exploration of the search space. The initial mean vector, m R", represents the initial est estim the search space, while the initial step size, (()), scales the search distribution. To ensure isotropic sampling at the outset, the initial covariance matrix, C(O), is typically set to the identity matrix, I. Note that it is common practice to scarch over normalized space, where the search distribution is isotropic and centered at the origin, to improve numerical stability. Hence, the initial mean vector and covariance matrix are initialized to zero and the identity matrix, respectively.
Algorithm 2: CMA-ES (Covariance Matrix Adaptation Evolution Strategy)
Input:
Population size X, initial mean (0), initial step size (0)) initial covariance matrix (()) I, weights w)...., Wy Output: Optimized solution m
Initialize:
Iog(uct () (og()))
Set. WN Σ,-, =1 (log(u+ }) log())))
for I.....ff
Normalize weights: wi
4. Set Ps <- 0, (0) <- 0, Hall +-
(0)
EE wx)² E w
Set learning rates co. Co. 01, ca. Ca, damping factor da, constant X vn(1 dn 2112)
Generation loop: for g= I. G-1 do
for k- to A do
Sample: X (imm
Evaluate fitness: f(x)) end
11
12
Sort XI....,XA XA by fitness, and select the p best solutions ml9+1) Σr= wix
Compute new
14
Update step-size evolution path:
(9+1) (1 cc))pp)) co co)Me C- & m(o+1)
15
olu)
16
he < (1p 1/11 (1 ex) (69t)) < (1.4 a41 Xx
17
Update covariance evolution path:
18
p!o+1) <- (1-c))pp
co)p!!) + ha cal2 co)Mett
ml+1) mlm) (9) ata)
19
Update covariance matrix:
20
C(++1) (1 ei C(s) (0t pe
(o+1)
co R wyy.y
(9:
21
where y; σ(
22
Update step size:
23
x(s+1) + (()) exp
Go
[p 11 -1))
X
24
end
25
return m (8-1)
20

---

Published in Transactions on Machine Learning Research (2/2025)
The table below summarized the hyperparameters used in the CMA-ES algorithm for the Pendulum and Quadrotor tasks in this paper.
Hyperparameter
Name
Pendulum
Quadrotor
G
Generations
40
100
Population size
200
200
Selected solutions
20
20
Ca
Step-size learning rate
0.40
0.65
Cc
Covariance learning rate
0.14
0.68
Rank-1 update rate
0.0024
0.19
Ca
Rank-u update rate
0.038
0.27
dg
Step-size damping
1.40
2.41
m (0)
Initial mean vector
0
0
o (
Initial step size
0.4
0.4
Initial covariance matrix
F Proximal Policy Optimization
Proximal Policy Optimization (PPO) is a reinforcement learning algorithm designed to optimize policies by maximizing the expected return (Schulman et al., 2017). However, reproducing PPO's results can be challenging due to its sensitivity to hyperparameters and implementation details (Huang et al., 2023). For this reason, we provide the exact PPO implementation used in this paper the supplementary material. Our implementation largely based on (Lu et al., 2022), which itself builds upon Huang al. (2022).
The table below summarizes the hyperparameter settings used for the two tasks in this paper:
Hyperparameter
Name
Pendulum
Quadrotor
T
Total timesteps
2 x 10
10 x 10
7
Learning rate
3.26 x 10
9.23 x 10
Tenvs Nsteps E
Number of environments
128
128
Number steps per update
32
64
Number of epochs
16
Aminibatch
Number of minibatches
16
8
7
Discount factor
0.9939
0.9844
AGAB
GAE lambda
0.971
0.939
Cclip
Clipping epsilon
0.164
0.131
Q'ent
Entropy coefficient
0.01
0.01
Avy
Value function coefficient
0,802
0.756
Jmax
Max gradient norm
0.963
0.76
Thidden
Number of hidden layers
Hanits
Number of hidden units
64
64
@hidden
Hidden activation
tanh
tanh
ind
State-independent action noise
True
True
Squash
Action squashing
True
True
Manneal
Anneal learning rate
False
False
Norm
Normalize environment
True
True
21


Reviews:
[{"review_id": "vaOyaxz3bN", "reviewer": "Reviewer_jGJv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper brings up an important problem in robotics and sim2real frameworks. Paper is well written and the library/framework would be a nice contribution to the robotics/ML community. I do have some questions about the experimental evidence and comparisons as below:\n\nQuestions/Improvement Points:\n- I am curious how this stacks up against other simulators which do not have this. Other simulators and briefly mentioned in the Sim2Real related work section which also support hardware acceleration but maybe does not model the latency dynamics of the different control loops/systems. For example, IsaacGym/Sim is commonly used now in many different papers and publications, especially in legged robotics where we have seen some really cool results. What is the comparison here? Why is it not important for them? It would be nice to discuss this.\n- I think it would significantly improve and strengthen the paper to show pictures/videos of the real-world experiments conducted, along with its outcomes to allow readers to better visualise the benefits of the simulator.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper contributes a framework/implementation to improve robotics simulation. In particular, the proposed framework contributes graph-based simulation model to more effectively model dynamics that come from communication and motor latency between various asynchronous systems in robotics. The goal is for this to lead to more effective sim2real policies. Additionally, the framework is built on Jax, Brax and aims to best utilise parallelzation and hardware accelerators.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No major changes. But if you could attach some video/picture evidence of real-world experiments conducted which better visualise the difference between a policy trained with REX vs not trained with REX and its siginicance in the real-world, that would be great.", "broader_impact_concerns": "No major concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zPmkGftj62", "reviewer": "Reviewer_6SbQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The introduction of a graph-based simulation framework (REX) that explicitly models latency, asynchronous dynamics, and hierarchical systems is a significant contribution. I am unable to judge the novelty of this approach. The framework successfully bridges the sim2real gap by addressing challenges in system delays due to communication, computation, sensing and actuation.\n2. The use of Gaussian Mixture Models (GMMs) for stochastic delay modeling and Smith Predictor-based estimators for delay compensation is interesting.\n3. The framework supports integration with multiple physics engines (e.g., Brax, MuJoCo), making it adaptable for various robotic systems.\n4. Demonstrated effectiveness on two distinct real-world robotic systems: pendulum swing-up and quadrotor path following. Experimental results validate improvements in sim2real transfer and showcase the framework's ability to identify and compensate for delays accurately.\n5. Effective use of hardware acceleration through JAX for both training and runtime execution.\n\n**Weaknesses**\n\n1. The paper does not lay out details about GMM modeling. How sensitive is downstream performance to the accuracy of this learned model? What are the minimum data requirements to train this model?\n2. Modeling hidden delays requires specifying a maximum and minimum bound for each delay.\n3. Usage of multiple out-of-the-box components, details of which are not included in the paper - e.g. the lightweight dynamics model (Dernet et al., 2020) and CMA-E Strategy used for parameter optimization. Having the required background details (e.g. in the appendix) makes the paper self-contained, which I believe is important in a good paper.\n4. What are v_path and e_path in Table 2? Missing exact definitions of these error metrics.\n5. While the experiments seem comprehensive, they are limited to controlled environments (pendulum and quadrotor tasks). Extending the validation to more diverse or complex real-world robotic systems could improve the generalizability of the framework. \n6. The graph-based model’s scalability to extremely large or highly complex systems (e.g., multi-agent setups or industrial robotics) is not fully explored. Providing insights into potential bottlenecks and solutions for large-scale systems would be useful.\n7. While comparisons with existing frameworks like EAGERx and Drake are mentioned, a more detailed quantitative evaluation against these alternatives in terms of performance, accuracy, and scalability would strengthen the claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a framework, REX, designed to bridge the gap between simulation and real-world robotic systems. The contributions and key aspects of this work can be summarized in the following 3 categories:\n\n**Introduction of REX framework**\n1. A graph-based simulation model optimized for accelerator hardware.\n2. Explicit modeling of asynchronous dynamics, measurable and hidden delays (in communication, computation, sensing, and actuation), inherent in real-world robotic systems.\n\n**Integration with advanced tools**\n1. Leveraging JAX for parallelized computation, automatic differentiation, and high-performance training.\n2. Compatibility with multiple physics engines (e.g., Brax, MuJoCo) within the graph-based model, allowing modular and extensible simulations.\n\n**Experimental Validation**\n1. Demonstrated the framework’s capabilities on two real-world systems: a pendulum swing-up task and a quadrotor path-following task.\n2. The pendulum experiments highlighted the framework’s ability to estimate system dynamics and delays, achieving accurate sim2real transfer with delay compensation techniques.\n3. The quadrotor experiments showcased scalability to higher-dimensional systems and emphasized the critical role of delay-aware training.\n\nThis paper's key contribution lies in its comprehensive approach to addressing latency and dynamics mismatch in sim2real systems. By combining robust delay modeling, system identification, and efficient parallelized computation, it aims to advance the sota in robotic learning and simulation frameworks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Including details about GMM modeling, and analysis on sensitivity to downstream performance. Analysis on minimum data requirements to train this model would be a plus. (critical)\n2. Including details of different components included in the paper (dynamics models and optimization strategies used in the experiments). (critical)\n3. Discussion around potential scaling-up to more complex real-world robotic systems. (strengthen)\n4. Comparisons with existing frameworks like EAGERx and Drake, potentially a more detailed quantitative evaluation against these alternatives. (critical)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mh06EPhHCb", "reviewer": "Reviewer_qxE9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The experimental evaluation is convincing. Authors present results on two sim2real transfer tasks.\n- The paper reads very well and feels polished. Figures are of high quality. \n- The authors submitted their code to reproduce their experiments.\n\n**Weaknesses:**\n\n- My main point of criticism is the quality of the code repository. Since this paper revolves around an implementation, the code repository is a central component of the submission and should therefore be of high quality. The following points regarding the repository should be addressed before publication:\n    1. The source code is mostly contained in two unstructured folders. I would suggest that the authors refactor their code and split it into two parts: (i) general code that can be used by the readers to solve their sim2real problems and (ii) code that is specific to the experiments presented by the authors (e.g., the contents of the notebooks folder or the pendulum folder).\n    2. There is only a minimal documentation. I would expect a polished documentation of the functions and classes that future users can use as reference. I would suggest that the authors write a documentation using pydoc or similar documentation generator.\n\n    3. I appreciate that there are some instructions that guide a potential user of the repository to solve their problem with the REX package, however those instructions should be more prominent within the repository/documentation.\n\n    4. There are no tests. I suggest the authors write some unit tests (e.g. using pytest) that help maintaining the repository.\n\n    5. Please follow the PEP 8 code style.\n\n- Minor points: \n\n    1. Code shown in Figure 3 (c) does not follow PEP8 code style. I would suggest to remove the unnecessary whitespaces.\n\n    2. Computational complexity is not the right term when referring to a computational runtime analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenges of sim2real transfer caused by unmodeled dynamics and latencies in real-world robotic systems, proposing a novel framework called REX to incorporate latency effects and optimize for parallelization accelerator hardware. The framework is validated on two systems: the real-world pendulum swing-up task and a quadrocopter path following task. The authors demonstrate the effectiveness of their REX framework in improving sim2real performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Improve the REX repository. \n- Please, clarify the error metric $v_\\text{path}$ and $e_\\text{path}$ used in Table 2.\n- Address the minor points.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are satisfied with the authors' responses and improvements over the original reviews and feedback, so I am recommending that the paper be accepted as is.
Decision: Accept (poster)