Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks

Authors: alex  quach
makram  chahine
alexander  amini
ramin  hasani
daniela  rus
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-04 21:37:16

Abstract: Simulators are powerful tools for autonomous robot learning as they offer scalable data generation, flexible design, and optimization of trajectories. 
However, transferring behavior learned from simulation data into the real world proves to be difficult, usually mitigated with compute-heavy domain randomization methods or further model fine-tuning. We present a method to improve generalization and robustness to distribution shifts in sim-to-real visual quadrotor navigation tasks. To this end, we first build a simulator by integrating Gaussian Splatting with quadrotor flight dynamics, and then, train robust navigation policies using Liquid neural networks. In this way, we obtain a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, crafty programming of expert demonstration training data, and the task understanding capabilities of Liquid networks. Through a series of quantitative flight tests, we demonstrate the robust transfer of navigation skills learned in a single simulation scene directly to the real world. We further show the ability to maintain performance beyond the training environment under drastic distribution and physical environment changes. Our learned Liquid policies, trained on single target maneuvers curated from a photorealistic simulated indoor flight only, generalize to multi-step hikes onboard a real hardware platform outdoors.

Main paper: 
 introduction: 
 Recent breakthroughs in simulation now allow for the creation of more lifelike and varied datasets. Techniques like neural scene reconstruction [1, 2, 3] enable detailed, photorealistic 3D models that mirror real scenes, providing advanced training grounds for robotics. Generative AI also expands training data possibilities, with entirely new, realistic environments and scenarios created from text descriptions or similar inputs [4] .Building on these advances, we design a methodology to improve the generalization and robustness of autonomous quadrotor navigation from simulation to real-world environments. Our approach includes three schemes: 1) integrating a 3D Gaussian Splatting radiance field with a quadrotor flight dynamics engine to create a realistic simulator for generating high-quality training datasets for drone navigation; 2) designing an imitation learning pipeline to train liquid time-constant (LTC) networks [5, 6] , which have out-of-distribution generalization capabilities [7] , on the generated data; and 3) deploying the trained networks on a real drone to demonstrate robust generalization.Quantitative flight tests show successful transfer of learned navigation skills from simulation to real-world settings. Policies trained exclusively on photorealistic simulated data of indoor flights targeting single manoeuvres generalize to complex, multi-step tasks in outdoor environments on real hardware. To the best of our knowledge, this is the first published report of an autonomous quadrotor policy trained entirely in simulation using imitation learning, without scene randomization, that is capable of being deployed into the real world, and generalizing to out-of-distribution environments. Our contribution relies on five key pillars:1. Dynamics Augmented Gaussian Splatting Simulator: A simulator that combines the strengths of Gaussian Splatting and flight dynamics, providing both photorealistic rendering alongside high-fidelity dynamics.2. Implicit Closed-Loop Augmentation: Expert trajectory design that inherently includes edge cases and recovery scenarios, eliminating the need for additional closed-loop augmentation and model fine-tuning.3. Liquid NNs Augmentation with Irregularly Time-Sampled Data: Robustifying performance of Liquid NNs via irregularly time-sampled data training, and enhancing their adaptability to diverse temporal dynamics.4. Simulation and Real-World Validation: Extensive testing to validate the effectiveness of our method across different simulation environments and in real-world flight scenarios.5. Zero-Shot Real-World Transfer: Showcase of the pipeline's capability for zero-shot transfer, enabling a quadrotor to successfully navigate outdoor environments in a real-world hiking demonstration. 
related works: 
 Imitation Learning. Imitation learning, and more specifically behavior cloning, involves an agent learning a task by mimicking expert demonstrations, which is especially advantageous when defining a reward function is difficult. This approach captures implicit navigation knowledge in complex environments and is more sample-efficient than Reinforcement Learning (RL) [8] , though it can suffer from policy stationarity and compounding errors [9] . Robot navigation methods built on foundation models [10, 11] and diffusion policies [12] , exhibit very potent capabilities but violate compute constraints for real-time operability on edge devices.The generalization problem of few-shot, one-shot, and zero-shot imitation learning agents has received extensive attention in the literature, with methods including augmentation strategies [13, 14] , human interventions [15, 16] , goal-conditioning [17, 18, 19, 20] , reward conditioning [21, 22, 23, 24] , task-embedding [25] , and meta-learning [26] .We opt for this specific setting since expert drone trajectories are easy to design in simulation and thus labeled data can be produced flexibly at speed for specific flight tasks.Liquid Neural Networks. Liquid Neural Networks (LNNs) are a family of continuous-time (CT) neural networks [27] that can be trained via gradient descent in modern automatic differentiation frameworks. Their building blocks are Liquid Time Constant networks (LTCs) built off of leaky-integrator neural models [28] with the steady-state dynamics of a conductance-based nonlinear synapse model [29] . The model is a differentiable dynamical system with an input-dependent varying (hence liquid) time characteristic. Outputs are obtained by numerical differential equation solvers in the ordinary differential equations formulation and by continuous functions when approximately solved as Closed-form Continuous-time (CfC) models [6] . Amongst LNNs' desirable characteristics are their stable and bounded behavior, superior expressivity [5] , improved performance on time series prediction [30] and great promise in both end-to-end imitation and reinforcement learning, to map high-dimensional visual input stream of pixels to robust control decisions [7, 31, 32, 33] .Note: LNNs refer to the general category of models that are presented either by LTCs or CfCs. In this work, we use CfCs which have exhibited superior generalization performance when trained via behavior cloning for out-of-distribution flight tasks [7] .Sim-to-Real Transfer. Photorealistic simulators are popular for training quadrotor visual policies [34, 35, 36, 37 ]. Yet, photorealism is not sufficient to ensure the transfer of skills learned in simulation to the real-world, due to differences in physical properties, sensor noise, or actuator dynamics. Many sim-to-real techniques have been developed to mitigate the transfer challenge [38, 39, 40, 41] . Generalization to unseen environments is mostly achieved via domain randomization [40] , which incentivizes networks to learn representations that are scene and domain-independent by randomizing input features that are superfluous to the task. Indeed, amongst existing techniques [42, 43] are position randomization, depth image noise randomization, texture randomization, size randomization, and so on.In this work, we use a well-designed randomization in the positioning of task-relevant objects in the scene and perform RGB image augmentation on the training data (brightness, contrast, and saturation) [13, 44, 45, 46] . However, we do not resort to any randomization in the scenes simulated.In fact, we attempt to provide a simple environment data generation and training protocol and to subsequently test the fundamental task representation learning capabilities of networks.Gaussian Splatting. 3D Gaussian splatting (3D GS, or GS for brevity) [2] is a technique for explicit radiance field rendering. It relies on the fitting of millions of 3D Gaussians to the training dataset images, a significantly different approach from the neural radiance field (NeRF) methodologies [1] .Offering real-time rendering without visual quality compromises, GS enables many new avenues in fields such as virtual reality and augmented reality, real-time cinematic rendering [47, 48, 49] , but also autonomous driving [50] . The explicit GS construction comes with significant advantages, as it enables real-time rendering capabilities in addition to vast levels of control, compositionality and editability for complex scenario handling [51, 52] .Connections between Unmanned Aerial Vehicles (UAVs) and radiance field rendering models are well established, primarily focusing on optimizing UAV deployment for training these models. Notable methods include real-time online NeRF training from UAV camera streams [53] , efficient reconstruction of unbounded large-scale scenes [54] , and autonomous positioning for real-time 3D reconstruction [55] , demonstrating success in various scenes and structures [56] . Additionally, GS has been utilized in Simultaneous Localization and Mapping (SLAM) for active perception modules [57] and real-time SLAM algorithms [58, 59, 60] . The simulation of quadrotor flight in neural radiance field environments is explored in [61] , where planning and estimation algorithms are developed for deployment within the NeRF setting. Closest to our simulator is the PEGASUS system [3] , which combines a physics engine with GS rendering and enhances objects with physical properties for manipulation data generation. 
methods: 
 The end-to-end visual navigation model π takes sequences of RGB images captured from drone onboard camera as input. For training, each frame x i is labeled with the instantaneous quadrotor velocities and yaw angular rate in the quadrotor body frame. More precisely, its label, y i , is a 4dimensional vector containing the x-(pitch axis, forward/backward), y-(roll axis, left/right), and z-(throttle axis, up/down) axis velocities in body-frame, in addition to the yaw rate of the quadrotor. The behavior cloning process, depicted in Fig. 1 , trains the network, π, parametrized by θ to output at time t a control command, µ t , given an input image, x t , that minimizes a Mean Squared Error (MSE) loss function relative to the ground truth labels.GS model of the indoor flight room. The environment selected for rendering is an indoor fight room. Images are collected using an iPhone 12 Pro camera via the Polycam app, resulting in a dataset of 430 images at 1024 × 768 pixels. The images are processed with COLMAP to create depth maps, and the GS model is trained on these outputs at half resolution, rendering images with satisfactory photorealism (Fig. 2 ) despite some visual artifacts.Adding models of task-related objects. We use two objects-red and blue spheres-defined in the geometry object file format (obj). Using the open-source 3D creation suite Blender, we render images of the spheres from all angles. We then apply the same training procedure as before to train a Gaussian Splatting model for each object. Finally, we manually tune the size and intensity of the trained GS models to ensure smooth blending into the indoor environment, as shown in Fig. 2 .Dynamics for GS trajectories. To generate images reflecting the dynamics of a simulated quadrotor, we align the visual GS world with the dynamics solver by finding the gravity-aligned vertical vector. Using the PyBullet physics engine [62] , we tune the scaling to ensure consistent object positioning between the environments. The physics simulation runs at 240Hz, while inference occurs at a slower rate. The GS camera moves based on relative displacement in PyBullet, rendering updated views from the quadrotor's positions for trajectory data or neural network inference (see Fig. 3 ). Implementation details are in Appendix A.The task is to navigate through multiple checkpoints using visual cues. The agent must reach the current target and turn according to its color: left for red and right for blue, as shown in Fig. 2 . This setup mimics following trail blazes on a hike, guiding the hiker both in direction and toward the next cue. Consequently, each expert demonstration trajectory consists of two phases: approach and turn. We generate an equal number of right and left trajectories to avoid bias in the models.Approach. Naive Imitation Learning for autonomous navigation often fails when encountering trajectories outside the training distribution. To address this, recovery trajectories-showing how to return to desired paths from suboptimal positions-are added to the training data. This method is usually infeasible with real-world demonstrations but can be synthesized. Here, we ensure all trajectories contain recovery information by positioning the quadrotor at trajectory starts so the target partially appears at the image border and by randomizing the initial target distance. This exposes the network to various approach angles and distances, driven by PID controllers that adjust the quadrotor's velocities and yaw rate to center the target in the frame. The significance of this strategy is discussed in Appendix C.4.Turn. The turn procedure is straightforward and requires no special treatment, as it begins once the drone hovers at a fixed distance from the target. We send a constant yaw rate command, with an amplitude of about 12 • per second-nearly double the maximum rate during the approach phase when the sphere is at the frame's edge. This clearly distinguishes the switch in behavior at test time, reducing the risk of confusing a segment of the approach maneuver with a decision to turn. first an approach where the drone gets close to the target and centers it in the middle of the image, followed by a turn procedure in the direction corresponding to the color of the target.It is often assumed that the inference frequency of a neural network at test time is that of the training data. This holds when data collection and model deployment are both executed on the same platform and data processing and inference times are negligible with respect to the sample period. In fact, popular recurrent models cannot handle data that is irregularly sampled or sampled at a different frequency, as they treat time steps between inputs as a constant during training.Neural ODEs [63] , however, can be evaluated at arbitrary time intervals, as their hidden states x(t) evolve according to the time-continuous differential equation, dx(t) dt = f (x(t), I(t), t, θ). More specifically, the closed-form CfC representation of the hidden states is given by [6] :EQUATIONHere, x(t) is the hidden state, ⊙ is the Hadamard product, σ is the sigmoid function, f , g, and h are three neural network heads with a shared backbone, parameterized by θ f , θ g , and θ h , respectively. I(t) is an external input, and t is time sampled by input time-stamps, as illustrated in Fig. 10 in Appendix D.As time appears explicitly in equation ( 1), the formulation eliminates the need for complex numerical solvers (speedup between one and five orders of magnitude). We utilize this flexible data handling to implement an irregularly sampled data augmentation scheme that randomizes the time steps between training samples, helping networks generalize across frequency domains. 
hardware platform: 
 All real-world evaluations of trained policies are executed on a DJI M300 RTK quadcopter. The M300 interfaces with the DJI Manifold 2 companion computer, enabling programmatic control of the drone. The onboard computer runs an NVIDIA Jetson TX2, which has GPU capability for onboard inference in real-time (more information about the hardware setup in Appendix A). and are trained on the same collection of trajectories. However, the frequencies at which the data points are logged along the trajectories are modulated to generate three different datasets.The first model, called Liquid, is designed for time-scale augmentation. The frequency of recorded images and labels varies throughout the trajectories, with each data point taken at a random time difference corresponding to frequencies between 1 and 10 Hz (see Appendix B). The dataset includes input images, labels, and the time delta since the last input, as discussed in section 3.3. Liquid 3Hz and Liquid 9Hz are trained on datasets with constant sampling rates of 3Hz and 9Hz, respectively.LSTM Baseline Model. Long Short-term Memory (LSTM) [64] is a discrete-time neural network that consists of a memory cell that is controlled by learnable gates that access, store, clear, and retrieve the information contained in the memory. Amongst a plethora of RNN architectures tested on similar visual flight navigation tests, the LSTM architecture seems to offer the best performance amid the non-liquid neural networks [7] . Thus, we select it as our baseline and train on the 3Hz dataset to maintain proximity to the test hardware running frequency (LSTM cannot handle irregularly sampled data). The LSTM architecture we devise comprises just over 300k parameters. It is worth noting that its size is about 25 times that of its liquid counterparts that we deploy in this work.Evaluation protocol. Performance is associated to manoeuvre success rates. For a checkpoint to be valid, we require the quadrotor to come to a hover at a distance corresponding to the stopping position seen in training (0.5 m in simulation, between 1 and 1.5 m for the real platform) and then turn in the correct direction at a rate close to that used of the labels (above 10 GS-to-Real. The sim-to-real transfer evaluation, in the indoor flight room used for training the GS rendering, is deployed on the Matrice 300 quadrotor platform. We use cardboard disks as the goals. In turn, we position the targets on a tripod in the center of the room at a height of about 2 meters. We select five starting positions around the target, placing the quadrotor 4 to 5 meters away to give each agent four attempts at each color and expose them to multiple room angles. The translation velocities inferred by the networks are scaled in such a way as to balance enough time for human intervention and repeated test efficiency. The per color and total success rates are given in Table 2 .The challenge of direct sim-to-real Transfer is epitomized by the large drop in performance suffered by the LSTM agent (from 0.72 in GS to 0.125). Liquid neural networks, on the other hand, suffer minimally in comparison. Indeed, both the Liquid 3Hz and 9Hz agents succeed just over half of the time. Again, it is the time-scale augmented Liquid agent that performs best. It goes from 96 to 75% success rate, maintaining balanced performance across colors.GS-to-Real + Generalization. We deploy the same test protocol on an outdoor lawn in an urban campus. In addition to the completely unseen scene, the various starting positions now expose the agents to sunlight from various angles, making for a challenging generalization scenario. We compare the best Liquid model and the LSTM model for reference in this setting, and the results are provided in Table 3 .Unsurprisingly, the LSTM agent is incapable of completing the task even once in 40 attempts. Our Liquid agent, however, demonstrates zero-shot task completion capability. Though relatively struggling with red targets (40%) in the outdoors setting it does better on blue checkpoints (60%), exhibiting the ability to perform the simulation-learned task more than half of the time completely out-of-distribution outdoor real-world setting. Challenging background lighting conditions, as well as tree shades and reflections off buildings impact the appearance and robust perception of targets.Success rate Red Blue Total Liquid 0.4 0.6 0.5 LSTM 0 0 0Table 3 : GS-to-Real zero-shot evaluation results of models on single target runs on the outdoor campus lawn (N = 40).Encouraged by these results, we design a multistep outdoor setup in an alternative campus location to showcase performance on the comprehensive hiking task. Our Liquid agent demonstrates state-of-the-art transfer and generalization by successfully navigating through a series of 5 consecutive checkpoints. Videos of training data runs, experiments and real world demos are provided on the project website (link). 
limitations: 
 Dynamic scenes: Gaussian Splatting, like other neural rendering techniques, excels in static scenes but struggles with dynamic environments, such as moving objects and changing conditions, limiting its effectiveness in generating dynamic data for real-world scenarios. While affine transformations can adjust Gaussian attributes for moving objects, this process increases rendering time and is constrained to discrete components. In contrast, Liquid neural networks demonstrate robustness to dynamic backgrounds during testing [7] , allowing them to focus on tasks despite moving distractions. However, efficiently learning from dynamic data in applications like urban air mobility and wildlife tracking remains outside the scope of this work.Trajectory augmentation: As tasks become more complex and require longer planning horizons, designing efficient trajectory augmentation schemes becomes increasingly challenging. Current methods may not cover the full range of scenarios, especially edge cases and recovery situations, which are crucial for robust performance. This general limitation of imitation learning holds for datasets collected in the real world, in simulation, and in rendered environments. Distribution shifts: Large distribution shifts between training and real-world environments can severely hinder performance; thus, the need for domain randomization to bridge substantial gaps remains essential for robustness across real-world environments not encountered during training.Addressing these limitations is crucial for improving the robustness and applicability of visual endto-end autonomous navigation systems trained in simulation for diverse real-world scenarios. 
conclusion: 
 3D Gaussian Splatting has emerged as a potential solution for generating realistic visual training data flexibly and at scale for autonomous systems, surpassing the limitations of real-world data collection. However, bridging the gap between these simulated training environments and real-world applications remains a significant challenge in robot learning, typically addressed through domain randomization. In this work, we propose an end-to-end protocol for training autonomous agents for deployment into the real world, encompassing data-driven simulation methods, and reliance on Liquid neural networks, with newly explored augmentation techniques for improved transfer and generalization. Comprehensive experiments empirically confirm the ability of learned agents obtained via our pipeline to directly be deployed on an autonomous quadrotor platform to successfully perform a hiking task in environments they have never seen before. We believe our approach represents a leap in the employment of Gaussian Splatting and Liquid neural networks for training robust visual end-to-end autonomous navigation policies for direct, real-world deployment. 
build realistic simulator train liquid neural net: 
 Zero-shotTransfer to real world 𝜋 ! (𝑥 " )Label : 𝑦 " = 𝑣 " # , 𝑣 " $ , 𝑣 " % , 𝜓 ̇"Input : = 𝑥 " Train 𝜇 "Gaussian Splatting Deploy Figure 1 : We create a photorealistic policy learning environment by combining Gaussian Splatting with quadrotor flight dynamics to generate high-fidelity training datasets. A model, π, with parameters θ, is trained using input images x t to match its predicted output µ t with ground truth labels y t = [v x t , v y t , v z t , ψt ], where v t denotes desired triaxial velocities and ψt is the yaw rate. Liquid neural networks learn the task through behavior cloning and achieve sim-to-real transfer. 
 
 Appendix: A Simulator and hardware setup A.1 Indoor room GS modelImages of the room are collected using a consumer smartphone (iPhone 12 Pro) camera through the free access application Polycam. The application automates the capture of frames, recording images when the novelty in the frame is sufficiently large. We thus obtain a dataset of 430 images of the room at a resolution of 1024 × 768 pixels. The images are processed with the generalpurpose Structure-from-Motion (SfM) pipeline COLMAP [65] to create depth maps. The GS model is trained on the COLMAP outputs at half resolution (512 × 384), densification interval of 500, and 30,000 iterations. Training takes around 15 minutes on a single GPU (NVIDIA GeForce RTX 3080 Ti). The obtained model renders images of satisfactory photorealism as seen in Fig. 2 , however, with the presence of many objects in the room and limited training, there remains some fogginess and some visual artifacts in the images. The purpose of this work is to achieve robust generalization, rather than to improve radiance field rendering. Thus we did not pursue further improvements of the GS model.We ensure that the points of view we use for the generation of images are on trajectories true to the dynamics of an accurately simulated quadrotor model. To do so, we first use the GS renderer user interface to approximately find the gravity-aligned vertical vector. This is necessary to align reference frames between the visual GS world and the dynamics solver. PyBullet physics is used in our simulator which we adapt from the codebase of the authors in [62] . The drone dynamics we use are based on Bitcraze's Crazyflie 2.x nano-quadrotor.Scaling is tuned manually such that distance units in PyBullet and units in the corresponding GS environment match. Furthermore, we ensure that the positioning of objects matches: for a given position and attitude, the drone front image extracted from the PyBullet environment and that obtained from the GS rendering contains the object of the same size and in the same position (as depicted in Fig. 3 ). The simulator thus behaves as a direct photorealistic enhancement of images seen from trajectories in the PyBullet physics simulator. Hence, each trajectory can be visualized in both environments and datasets can be generated for either.The physics simulation runs at 240Hz, and we allow low-level flight control to occur at the same frequency, although inference runs at a much slower rate. Indeed, for inference or data collection, we simulate the evolution of the system with constant velocity command for a number of steps corresponding to the desired period. Then, we compute the relative displacement in the PyBullet environment and convert it to the GS space's dimensions. Next, we move the GS camera accordingly and render the view from the updated quadrotor position and attitude. Finally, the captured image is added to the trajectory data or fed into the neural network for inference before repeating with the current velocity command (process visualized in Fig. 3 ).The DJI Onboard SDK (Software Development Kit) and its associated ROS wrapper provide an interface for feeding the drone's low-level flight controller with desired high-level translation velocities and yaw rate commands. The flight controller itself is a black box system provided by the manufacturer which controls the four-rotor speeds to track the velocities specified by the TX2 computer. It is worth mentioning that the dynamics of the platform we use do not match those of the nano-quadrotors simulated. A Zenmuse Z30 camera and gimbal are mounted on the underside of the drone, pointed forward. The gimbal compensates for the drone's current orientation, such that the roll or pitch of the drone does not result in the roll or pitch of the camera image. The gimbal follows the drone's current yaw such that the camera is always pointed forward. Input images gathered by the gimbal-stabilized camera are available to the companion computer via the SDK. We reach a runtime frequency of around 2.6 Hz with our setup, a budget shared between image capture, communication with the onboard computer, pre-processing and model inference, and sending of the output command.Designing our dataset, we limit the trajectories to the case of reaching and turning at a single target, as opposed to generating runs hopping through multiple checkpoints. This choice is deliberate as it gives us more control over the initialization augmentation procedure, leading to better training data efficiency while ensuring the multi-step capability of the networks through sequential composition.In the simulator, we always assume that the target is positioned at the origin of the (x, y) horizontal plane, and at a fixed altitude. Each new trajectory is generated using PID controllers that stabilize the yaw, altitude, and distance gaps to a centered position facing the target. We uniformly sample an angular position anywhere around the origin, a starting distance from the target in a prefixed range.We then randomize the initial attitude and altitude of the drone as to randomize the placement of the target in the first frame. Three quarters of trajectories are such that the target is at a lateral edge (vertical position uniformly sampled). In the remaining runs, the target is at a vertical edge, with uniformly distributed yaw. This procedure is summarized in Algorithm 1 and a visualization of the positions of the spheres in the first frame of a 100 sampled trajectories is provided in Fig. 5 . # Initialize roll and pitch to zero: (θ 0 , ϕ 0 ) = (0, 0) # Sample initial angle and distance:θ ∼ U(0, 2π) , d ∼ U(d m , d M )# Position the drone in the horizontal plane:(x 0 , y 0 ) = (d cos(θ), d sin(θ))# Sample chance parameter: c ∼ U(0, 1) # Select yaw and altitude accordingly:if c < 0.75 then z 0 ∼ U(z m , z M ) ψ 0 ∼ B(0.5, {-ψ M , +ψ M }) else z 0 ∼ B(0.5, {z m , z M }) ψ 0 ∼ U(-ψ M , +ψ M ) end if return s 0 = [x 0 , y 0 , x 0 , θ 0 , ϕ 0 , ψ 0 ]We also provide sample trajectories for 600 approach phases generated according to the above described protocol with the ground-truth informed PID expert controller in Figure 6 and Figure 7 . The trajectories show a dense coverage of the entire physical space available between the drone and the target in terms of both lateral and vertical displacements, thus including recovery information from a large set of situations by design.The time deltas are sampled to be divisible at the resolution of our simulation frequency (240Hz). Time delta in seconds, δt, is sampled by the following distribution:EQUATIONwith δt min being the physics simulation unit time step, N (µ, σ) a Gaussian distribution of mean µ and standard deviation σ, and α, β, µ 1 , σ 1 , µ 2 , and σ 2 constants tuned to ensure adequate training run lengths (we require a minimum of 64 frames for a single trajectory), while still representing the full range of values between 1 and 10 Hz. The probability mass function can be seen in Figure 8 . Situations where the drone shoots past the targets (no slowing down), turns in the wrong direction (turn triggered at the rates of interest but with the opposite sign), stays still for prolonged periods (sub centimeter per second forward velocity during the approach phase for over 20 seconds in the real world tests, or if taking more than twice the average time of a manoeuvre in simulation), or loses the target before completing the approach (target out of frame during the approach and/or approaching another visual cue), all constitute failure cases. In our experiments, we notice that the most common mode of failure is the loss of the target out of the field of view during the approach phase (Sim, GS and Real) as well as turns in the wrong direction after a successful approach (GS and Real). Cases where the agents get stuck indefinitely occurred a handful of times across testing platforms.The successful transfer of task completion capability from the GS trained liquid models to the real world poses the question of the level of photo-realism required to sustain such a transfer. Thus, we test whether models trained with non-rendered simulation data can generalize to the GS based simulation evaluation. We see this Sim-to-GS as an intermediate step between Sim-to-Sim and Sim-to-Real, which would most likely be informative about the prospects of achieving the latter. The results presented in Table 4 shed light on the difficulty agents have in generalizing meaningful behavior to environments with drastic visual characteristic changes. In fact, the Liquid 3Hz and 9Hz models suffer from a massive color-blind biases towards right turns and LSTM towards the left. The Liquid agent fails to initialize turns which leads to an extremely low performance. Thus we verify our claim that photo-realism is crucial for transferring vision-only end-to-end tasks learned in simulation. Elsewhere, we rerun the GS-to-GS experiment after halving the time budget allocated to the agents to complete the 2-step hike. This allows us to evaluate the speed and efficiency closed-loop navigation exhibited by the different networks. The results, provided in Table 5 , show the superiority of the Liquid agent which performs at the same standards in half the time (from 1.72 to 1.66 on 2target score), while the LSTM performance is halved on the 2-target metric (1.04 to 0.51) and drops significantly for the first target as well (0.72 to 0.51). Both regularly time sampled Liquid models fail to complete the second passage in the reduced time (apart from 2 full runs completed by Liquid 3Hz).We compare 5 trained instances of the Liquid model with the data generation scheme described in the paper (Full Window), 5 instances on a dataset generated with the same procedure but with the window (along which initial positions are sampled) halved in height and width (Half Window), and 5 instances on a dataset generated by randomly sampling the original yaw and height as to have the It is clear that sampling initial positions that never reach the sides and corners of the initial image is not a good strategy (even here, where testing does not involve extreme recoveries). Two training instances are around the 10 average hike length with the Half Window recovery dataset, with the others considerably worse. Our Full Window recovery strategy generates the best-performing models overall, although other training instances lead to poor performance. Uniform Sampling produces slightly less high-scoring models as well, with an improved average due to a smaller drop-off for lesser runs.We then set up a recovery-specific test, where we force the initial placement of the target in one of the four corners of the frame, and run 100 single-target reach and turn tests with a fixed time window for each training iteration as above (discarding the disqualified Half Window strategy). The success rates are given in Table 7 . We notice that while uniform sampling can only lead to a single model performing above the 80% mark, our full window approach only leads to a single checkpoint that does not reach that figure with all others well above and two out of five with perfect performance. Overall, this leads to an 18% improvement in performance on average and highlights the importance of maximizing recovery trajectories in the training dataset to ensure robustness.To assert the architectural superiority of liquid neural networks we modify the LSTM architecture to include the irregularly sampled time difference to the last frame as an additional input. We call this variant LSTM+δt. We train 5 instances of the LSTM+δt (with the LSTM hyperparameters) to compare to 5 retrained Liquid models on the PyBullet simulator data (same checkpoints as above).The models are all tested on the Sim-to-Sim 100 step hike. The average hike length obtained among the 10 initializations used in Section 4.3, as well as their average across training instances, are provided in Table 8 . . This shows that augmenting models with timestep knowledge is indeed beneficial, but much larger performance discrepancies are to be attributed to model architecture.Figure 9 provides a top-down view of the drone's trajectory in the horizontal plane during the demonstration provided in our supplementary material (video with 5 checkpoints in the completely out-ofdistribution environment). The blue and red dots represent the approximate reconstructed locations of the targets along the hike. We use a colorbar to visualize the variations in forward velocity v x . The trajectory starts in the bottom left corner and the drone makes its way up to the top right. We notice that as programmed in the training data, the forward velocity is relatively high during approach and then reduces to a near stop before turning in front of the targets. Although robust in navigating in a similar fashion to the proportional gain controller from training (increasingly aligned with target, no systematic overshoot), the smoothness and duration of transitions are somewhat variable due to control and time delay errors, exterior conditions (wind), and the high level of distribution shift that the model has to deal with. (Adapted from [6] with the authors' permission.)We balance the number of training epochs and the number of trajectories with the different frequencies by ensuring all models are trained with roughly the same number of training batches. We use 200 as the minimum number of trajectories for the 9Hz model (100 red and 100 blue). We use 600 training trajectories for the 3Hz models to use roughly the same number of frames. However, since there is a nonlinear relationship between the number of batches and the length of a single trajectory, we train for twice as many epochs in the 3Hz models to get roughly the same total number of training batches within a training run.We use Tree-structured Parzen estimators (TPE) [66, 67] , a sequential model-based optimization algorithm, to tune a subset of hyperparameters for each of the models (some values are hand-selected to reduce the search space dimension). The Optuna library [68] handles the hyperparameter optimization, exerting early pruning of models with a validation loss higher than the median of previously tested models (n > 3) after 10 epochs. Tuning includes the learning rate (LR), LR decay rate, and model-specific hyperparameters such as backbone unit count and connection seed. The combination of hyperparameters achieving the smallest validation loss over 40 trials of 50 epochs is used for final training. Each architecture is trained 5 times with the optimized hyperparameters, and the best-performing checkpoint on the validation loss metric is then deployed for testing. Model and training procedure parameters can be found in Table 9 and Table 10 . We compare the performance of 5 instances of training of the each model on the PyBullet dataset. We test the checkpoints on a single target test where the initial placement of the target is one of the four corners of the frame to maximize difficulty and evaluate robustness. The success rate over 100 trials for each is presented in Table 13 .The Liquid model averages a 89% success rate with a 11.4% standard deviation. The lowest performing instance succeeds three quarters of the time whereas the best achieve perfect performance. This shows that the model's hyperparameter tuning leads to consistently good checkpoints although training initialisation and stochastic optimization do, as in most learned models, have an impact on an instance's performance. Compared to the Liquid 3Hz and 9Hz, we see that irregular timesampling with the Liquid 1-10Hz model leads to higher repeatability over multiple trials. LSTM does consistently well in-distribution over all training iterations.  


Decision: Strengths:
1. Demonstration on real hardware, clear ideas and very well structured presentation. Good work on providing a nice appendix with details and supplementary video
2. Connecting sim2real work to liquid neural networks is very novel and promising

Weaknesses:
1. Discuss limitations of the approach more thoroughly - this is a major weakness appearing in every review.
2. More ablation can be useful in understanding which part of the contribution is most critical
3. Study is done only on one type of task, would others work?
4. is it possible to generalize to more task types and out door / dynamic scenes?

Reviewer 5nTe was concerned about the lack of quantitative metrics on hyperparameter choices and repeatability. The authors addressed this by clarifying their selection process and committing to a systematic evaluation in the final version. The reviewer also inquired about the importance of recovery trajectories, which the authors acknowledged and are currently investigating quantitatively. Lastly, the reviewer was curious about the impact of higher sampling rates, and the authors explained the potential downsides, such as overfitting.    

Reviewer iDUo pointed out the lack of ablation studies and questioned the generalizability to dynamic scenes and complex tasks. In response, the authors clarified the existing ablations and discussed the limitations in handling dynamic scenes, along with potential future directions. The reviewer also raised a concern about the manual alignment of the Gaussian Splatting scale, which the authors acknowledged as a limitation but also highlighted the trade-offs with its benefits.    

Reviewer wnWZ considered the combination of existing methods simplistic and the baselines weak. They also questioned the necessity of realistic rendering and physics simulation. The authors defended their contribution as a complete training scheme and are working on stronger baselines. They emphasized the importance of realistic rendering for complex tasks and highlighted the advantages of Gaussian Splatting for creating customizable simulators. 

All reviewers suggest accepting the paper unanimously post rebuttal.

Reviews: review: title: title: Review of Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks
paper_summary: paper_summary: The authors seek to address sim-to-real navigation policy transfer through the introduction of a simulator augmented for photorealism with Gaussian splatting and leveraging the out-of-distribution adaptability of liquid networks. The authors demonstrate the efficacy of their proposed imitation learning pipeline both in simulation and with real-world tests.
main_review: main_review: ### Overview of review

Overall, I consider this to be among the better papers I have reviewed for CoRL and has a good coverage with contributions, experimentation and related discussion. The core ideas are rational and clearly stated, the experiments are sufficiently thorough and demonstrate the efficacy of the proposed methods, and the discussion, for the most part, helps readers digest the most salient points of the work.

### Strengths 

- Clear ideas, well structured presentation and relatively strong contributions.
- There is a logical progression of experiments which successfully demonstrates the soundness of the basic policy training procedure, followed by the Gaussian splatting augmentation and then finally the sim-to-real transference both in and out of distribution. This helps build confidence at each stage that the presented rationalizations correspond quantitatively with evaluation metrics.
- Demonstration with real hardware.
- While not discussed in the main body of the paper, the additional information in the appendices on environment and training designs offers useful insight and context of how the training pipeline works.
- This work is relatively novel - as the authors note, it is not the first to consider the problem of training control policies in simulated photoreal environments constructed radiance fields, but to my knowledge, an exploration of the application of liquid neural nets in such a problem is indeed novel. In a circumstance less constrained for space, I might have wanted to see more comparisons with different radiance field techniques, but I do not think that is strictly necessary to demonstrate the core utility of the training pipeline being discussed here - as I am reasonably convinced that the proposed liquid network-based control would be able to generalize similarly well even when trained with other forms of photorealistic simulation, given that there are not too many strange rendering artifacts (though that would probably trip up most contemporary methods).

### Weaknesses

- I had originally intended to call into question the repeatability of the methods presented here but It is evident from the appendices that the design decisions and hyperparameters selections were made based on multiple tests. The appendices do not however numerically quantify how performant each of the 'best' hyperparameters were, relative to others considered, nor how well-performing the 'best-performing' checkpoints from the different training runs were relative to other checkpoints from other runs. If there are reportable values to back the design choices made, they would help strengthen the paper as an independent work and would not rely on readers to trust in the general effectiveness of each of the component methods used.

- Table 1 indicates that the 100 test runs are split 53 red and 47 blue. I do not believe it is ever clearly addressed why the split is not 50/50. I suspect it has something to do with the stated biases that some methods have to different colors of targets, but I can't be sure. Unless I missed something, that aspect of the training procedure could stand to be more clearly explained.

- There is no clearly separated discussion on the limitations posed by the proposed methods. The appendices discuss some of the failure cases but the main body of the paper does not do much to address the method's major failure modes and bigger-picture limitations and what sort of paths future work may take to address them.
questions: questions: 1. Perhaps this is ignorance born from a limitation in my understanding of liquid networks, but it is surprising to me that the 9 Hz network performs poorly w.r.t. the 3 Hz network. I would have assumed that more rapid sampling might offer more opportunities for correction and therefore improved performance. Can the authors provide insight into why the faster sampling might contribute to worse performance? Based on appendix Fig. 8, it would seem that the dataset sampled for the 1 - 10 Hz main Liquid net has collectively more data that would exceed 3 Hz, so I would have expected the performance to skew closer to that of the 9 Hz net rather than 3 Hz. Evidently the data variation helps the main liquid net but I have not satisfactorily rationalized why the 9 Hz net would be worse. Would a 1 Hz net be better by extension?

2. How important are the recovery trajectories when constructing the training dataset? Based on the context of the paper, it seems like it would be important but it lacks a quantitative metric.
limitations: limitations: 1
review_summary: review_summary: Overall, I believe this is a good paper and I would vote in favor of its acceptance. There are a few details however where I'm not entirely clear on the paper's and authors' positions, thus limiting my recommendation to a weak accept for now.
title: title: Real-world simulator with Gaussian Splatting for Visual Rendering and Flight Simulator for dynamics for flight navigation
paper_summary: paper_summary: The authors proposed a simulator scheme that provides photorealistic rendering from Gaussian Splatting with high-fidelity flight dynamics to train imitation learning policies for real-world flight navigation. They utilized liquid neural networks with irregularly time-sampled data training to increase policy robustness. Quantitative real-world flight tests show the trained policies from the proposed simulator effectively transfer to real-world flight navigation in a zero-shot manner.
main_review: main_review: **Pros**

- Using Gaussian spatting allows task-specific and scene-specific photorealistic real-scene rendering, eliminating the sim2real gap visually. For flights, the flight dynamics simulator provides an easy expert data generator. 

- By utilizing liquid neural networks, the policy can be trained on irregularly sampled data temporally, so that the network can learn to generalize across different frequencies, which is very useful and critical for real-time policy for dynamic flight navigation.

- The authors report task performance, ablating network design (liquid networks vs LSTM), and training data frequency (3Hz, 9Hz, non-constant sampling rates), and show that for GS-to-GS/Real settings, the proposed method (non-constant sampling + liquid network) achieves the best performance. 

- Contains real-world experiment results with video visualization in the supplement video. Nice supplemental video!

**Cons**

- Missing limitations section.

- While the result of the proposed method is statically better than the baselines compared, the final absolute success rate for GS-to-Real of the proposed method seems a bit low given the task complexity (0.5). Any intuition for this number?

- There is no apple-to-apple ablation studies that ablates the performance effects coming from the photorealistic simulator, data sampling rate trained, or policy network choice. Thus it's hard to figure out which out of these system components are the most critical.
- It’s not clear how the current method would generalize (if so, how well) to dynamic scenes which is typical and important for flight navigation. 

- The alignment of the GS scale with the simulator is manual, which is costly to easily scale to different task scenes and environments. 

- To really make the case for the effects for GS for flight simulator rendering, the experiment of Sim-to-Real would be needed for a more apple-to-apple comparison. 

- The paper only has one type of task (flight navigation with approach and turn), thus it's unclear how the proposed simulator scheme generalizes to more flight tasks.


**Questions**

- The authors assume the task is composed of two stages, approach and turn phases. While these make sense and seem to work well for the flight navigation task this paper considers, curious about the generalizability of these decomposed phases to more complex flight navigation tasks that involve more cluttered environments that require tighter maneuvers and significant pitch and roll motions. It would be helpful for the authors to provide some discussion on this potential limitation.
questions: questions: - The authors assume the task is composed of two stages, approach and turn phases. While these make sense and seem to work well for the flight navigation task this paper considers, curious about the generalizability of these decomposed phases to more complex flight navigation tasks that involve more cluttered environments that require tighter maneuvers and significant pitch and roll motions. It would be helpful for the authors to provide some discussion on this potential limitation.

- If time allows, to really make the case for the effects for GS for flight simulator rendering, the experiment of Sim-to-Real would be needed for a more apple-to-apple comparison. 

- More visualization of the predicted and executed flight path in real world would be helpful
limitations: limitations: 1
review_summary: review_summary: Creative and smart integration of existing techniques from multiple fields: model architecture, neural scene reconstruction, and flight dynamics simulator, to leverage the best from different worlds to allow a higher-fidelity flight simulator with a policy model able to train from it. Idea for integration is great, but lacks extensive apple-to-apple ablations, and task diversity to make a confident prediction about the scalability of the method for more complex flight navigation scenarios.
title: title: Solid work, unsure about significance
paper_summary: paper_summary: The authors built a Gaussian Splatting reconstruction for an indoor lab, and use this GS to render the augmented trajectories (hard to collect such imitation trajectory in real world) so that imitation learning policy can be more robust due to seen more diverse data. They also adopted NeuralODE to deal with non-constant time step.
main_review: main_review: The paper is relatively well-written. One can tell from the amount of technical details given in the main text and appendices that the authors put much work in this submission to make it a solid research paper. Though this submission is mostly a combination of existing toolboxes (GS, NeuralODE), and the combination is rather "taking off-the-shelf" and simplistic, the successful use of GS and NeuralODE might be informative to peer researchers and practitioners. Other strengths include the thoughtful design of the dataset & augmentation scheme, including collecting data on reaching one target and testing on research multiple targets consecutively.

Weaknesses:
1. As mentioned, all existing methods and tools are used almost off-the-shelf, and the final system can be seen as simple, affecting the significance of the contribution.
2. The baselines feel quite weak -- see rebuttal questions below
3. The authors mention the method can generalize to outdoor scene even if only trained only in completely different indoor environment. This might actually be an indicator that realistic rendering is not that important anyways and maybe random real background might suffice. Authors are encouraged to think about more analysis on evaluating the usefulness of the background.
4. The task is rather simple, and there is only one task. One might be concerned how this framework works on more challenging tasks. In this simple task of reaching to red/blue target, a reasonable guess is that background, realistic or not, will not be that important in learning anyways (connects to my point 3)
5. Limitations are not discussed properly. There is only a section in Appendix discussing current failure cases, but for Limitations it should be much broader. Also the failure case section does not provide visuals (image/video).
questions: questions: - Can you design a baseline that takes timestep as input? Currently the LSTM baseline seems unfair since it cannot take the delta t as input. One simplistic baseline could be using a transformer, and for each frame of image you append a positional embedding using current t or delta_t.

- For a naive baseline of the Gaussian splatting, what about using random real indoor images as background instead of GS-rendered images? How will that affect outdoor performance? Further, what about putting the synthesized trajectories into high-quality renderers with environments, such as Habitat, or OmniGibson - that is, do we really need GS built from real images for this simple task? (This is different from Appendix C.2)

- What's the algorithm for imitation learning, just standard supervised learning (Behavior Cloning)? Please provide details of BC if tricks were necessary to avoid accumulative error and drifting.

- Do you really need physics simulation? Are you just using it to integrate the drone dynamics, which can be simply done by writing your own numerical integrator?
limitations: limitations: 1
review_summary: review_summary: More baselines and ablations ought to be done to gain a better understanding where this good performance and zero-shot generalization really comes from. Also the paper should showcase its effectiveness in more than one task and more environments. But as mentioned, the paper, with some revision, might be useful to other researchers.