Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control
Authors: Weidong Huang
Zhehan Li
Hangxin Liu
Biao Hou
Yao Su
Jingwen Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
world model
humanoid
Primary area: reinforcement learning
TL;DR: We enable large-scale simulation pretraining and efficient real-world finetuning for humanoid robots by scaling off-policy SAC for robust zero-shot deployment and using model-based adaptation to improve safety and sample efficiency.

Abstract: Reinforcement learning (RL) is widely used for humanoid control, with on-policy methods such as Proximal Policy Optimization (PPO) enabling robust training via large-scale parallel simulation and, in some cases, zero-shot deployment to real robots. However, the low sample efficiency of on-policy algorithms limits safe adaptation to new environments. Although off-policy RL and model-based RL have shown improved sample efficiency, the gap between large-scale pretraining and efficient finetuning on humanoids still exists. In this paper, we find that off-policy Soft Actor-Critic (SAC), with large-batch update and a high Update-To-Data (UTD) ratio, reliably supports large-scale pretraining of humanoid locomotion policies, achieving zero-shot deployment on real robots. For adaptation, we demonstrate that these SAC-pretrained policies can be finetuned in new environments and out-of-distribution tasks using model-based methods. Data collection in the new environment executes a deterministic policy while stochastic exploration is instead confined to a physics-informed world model.  This separation mitigates the risks of random exploration during adaptation while preserving exploratory coverage for improvement. Overall, the approach couples the wall-clock efficiency of large-scale simulation during pretraining with the sample efficiency of model-based learning during fine-tuning. Code and videos: https://lift-humanoid.github.io

Published as a conference paper at ICLR 2026
TOWARDS BRIDGING THE GAP BETWEEN LARGE- SCALE PRETRAINING AND EFFICIENT FINETUNING FOR HUMANOID CONTROL
Weidong Huang', Zhehan Li Hangxin Liu', Biao Hou", Yao Su', Jingwen Zhang
1 State Key Laboratory of General Artificial Intelligence, BIGAT School of Artificial Intelligence, Xidian University
ABSTRACT
Reinforcement Learning (RI. is widely used for humanoid control, with on-policy methods such Proximal Policy Optimization (PPO) enabling robust training via large-scale parallel simulation and, in some cases, zero-shot deployment to real robots. However, the low sample efficiency of on-policy algorithms limits safe adaptation to new environments. Although off- off-policy RL and model-based RL have shown improved sample efficiency, the gap between scaeesccae pretrain- ing and efficient finetuning on humanoids still exists. In this paper, we find that off-policy Soft Actor-Critic (SAC), with large-batch update and high Update- To-Data (UTD) ratio, reliably supports large-scale pretraining of humanoid loco- motion policies, achieving zero-shot deployment on real robots. For adaptation. we demonstrate that these SAC-pretrained policies can be finetuned in new en- vironments and out-of-distribution tasks using model-based methods. Data col- lection in the new environment executes a deterministic policy while stochastic exploration instead confined to a physics-informed world model. This separa- tion mitigates the risks of random exploration during adaptation while preserv- ing exploratory coverage for improvement. Overall, the approach couples the wall-clock efficiency of large-scale simulation during pretraining with the sam- ple efficiency model-based learning during fine-tuning. Code and videos:
https: httiiffffftuutiiiiiooooooo www.mooi...ittuu.iii.. github.io
( Large-Scals Pretraining
Pretraining
Buffer
Lagrangian Dynamics
(+f.
log of
ParalleEEn
Off polol SAC
(g
Predictor
transfer
Pretrained Polkcy
((b) Ae. slaae) 1,2, Tap?
Freeze & Rouse
(s a.)
Buffer
Collect
Physics-Informe World Model
SAC Policy
Buffer
Updato
Policy
(1i) Finetuning
Figure Large-scale pretralning and efficient FineTuning (LIFT) Framework. In stage (i), implement SAC in JAX to support large-batch update and high UTD, achieving fast, robust con- vergence in massively parallel simulation and zero-shot deployment to real humanoid in outdoor experiments. In stage (ii), pretrain physics-informed world model on the SAC data, combin- ing Lagrangian dynamics with a residual predictor to capture contact forces and other unmodeled effects. In stage (iii), finetune both the policy and the world model new environments while executing only deterministic actions in the environment. Stochastic exploration confined to roll- outs within the world model. This framework enhances both the safety and efficiency of finetuning.
"Corresondiigg Author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Learning generalist physical agents is a long-standing goal in AI, and humanoid robots offer broad task coverage among robotic embodiments. PPO (Schulman et al. 2017) becomes a mainstream baseline for humanoid robot control due to its robustness and fast wall-clock convergence in mas- sively parallel GPU simulation (Schwarke et al., 2025). However, when only limited real-world cross-task data can be collected, on- an-policy methods are disadvantaged because they discard off- policy experience (Haarnoja et al.] 201 8b. [illicrap et al.] 2015] Fujimoto et al.) (018). Moreover, even when PPO policy achieves zero-shot deployment in new environments, performance met- rics can degenerate (Gu et al., (2024) (e.g., poorer velocity tracking or reduced execution precision). These limitations highlight the need for frameworks that can reuse prior experience and adapt poli- cies efficiently in new environments.
Off Off-policy algorithms offer an appealing backbone for this paradigm because they can leverage re- played experience for sample-efficien finetuming (Lillicrap et al. 2015). However, these algorithms tend overfit to replayed data, which can bias exploration during fine-tuning, especially when large-scale parallel simulation with them has received limited attention. During fine-tuning, directly adapting a policy to a new environment often leads to unsafe actions because of stochastic explo- ration. This risk is especially high for humanoid robots: small support polygons, particularly during single-support phases, make the system highly sensitive to perturbations.
Constraining exploration within learned world model provides safer and more sample-efficient alternative (Levy et al., 2024, Hafner et al.] 2023), though synthetic rollouts may introduce model bias that destabilizes finetuning (Ha & Schmidhuber 2018 Janner et al., 2019)) A physics-informed world model (Levy et al., 2024) that incorporates structural priors from physeis improves the fidelity of synthetic rollouts, reduces model bias, and enables more reliable finetuning even with limited data. However, training such model-based systems from scratch (Levy et al., (004) remains ex- tremely slow in wall-clock time and prone local minima. Meanwhile, we find that off-policy SAC (Haarnoja et al. (018b) integrates more naturally with model-based algorithms (Levy et al.) 2024] than PPO, achieving robust and sample-efficient improvements. These findings further moti- vate our pipeline. We introduce Large-scale pretralning and efficient Fine'
Line'uuine Tuning (LIFT), a three- stage framework as shown in the Figure [ (i) large-scale policy pretraining, (ii) physies-informed world model pretraining, and (iii) efficient finetuning of the policy and world model. We make the following contributions:
1. We provide a scalable JAX implementation of SAC that supports robust convergence in massively parallel simulation and zero-shot deployment to a physical humanoid robot within one hour of wall-clock training on a single NVIDIA RTX 4090 GPU. The resulting SAC policy also serves as the policy module within our model-based finetuning stage.
2. We develop a fine-tuning strategy that executes deterministic actions in new environ- ments while limiting stochastic exploration in physicsinformed world model, which improves sample efficiency and stabilizes convergence. This physics-informed design is pivotal for finetuning, enabling data-efficient in-distribution adaptation and stronger out- of-distribution generalization.
3. We release an open-source pipeline for humanoid control spanning pretraining, zero-shot deployment, and finetuning, providing a practical baseline for the robotics community.
2 RELATED WORKS
Large-Scale RL Pretraining in Simulation. With the rapid progress of GPU technology, several GPU-based physies simulators have emerged, such as IsaacGym (Makoviychuk al. 2021). J Mu- joco Playground (Zakka et al.] 2025). and Brax (Freeman et al.] (021). These frameworks enable fast parallel simulation of thousands of robot environments directly on GPUs PPO (Schulman et al., 2017) has become the dominant baseline in this setting due to the ease of implementation and robust convergence. By leveraging domain randomization (Tobin et al. 2017). PPO policies can achieve fast wall-clock convergence and even zero-shot transfer to physical humanoid robots (Rudin et al.) 2022 Gu et al., 2024). However, the low sample efficiency of on-policy methods limits their ability to adapt or continue training in new environments.
2

---

Published as a conference paper at ICLR 2026
Off-Policy RL. When data collection is expensive risky, improving sample efficiency becomes critical. pollcy algorithms such as TD3 (Fujimoto et al. (0188) and SAC (Haarnoj et al. (018b) achieve higher sample efficiency by reusing past experiences and leveraging critic gradients to up- date the policy. SAC has been used to train quadruped rohots from scratch with limited real-world data (Haarnoja et al. 2018e; (Ha et al. 2020 Haarnoja al.) [0188a (mmihh et al. (022b), and pre- trained SAC policies can be finetuned on hardware with the same algorithm (Smith et al. |202aa] Ball ct al. 2023). However, these methods typically rely on stochastic exploration in the environment, where injected action noise may damage actuators or induce unsafe states. This risk is amplified for humanoids, whose smaller support polygons (especially in single-support phases) make them sensi- tive to perturbations compared to quadrupeds. These methods also underutilize large-scale parallel simulation, resulting in high wall-clock training time. Recent large-scale off off-policy efforts include Parallel Q-Learning (Li et al. 2023), which scales DDPG (1 (Lillicrap et al. (0155) across massive sim- ulations yet without sim-to-reat validation, and Fas (Seo et al. 2025). which achieves humanoid sim-to-real but its finetuning ability in new environments remains unclear. Raffin (2225) show that SAC can be made stable in massively parallel simulators through extensive parameter tuning, but their work also lacks sim-to-real evaluation and does not examine fine-tuning performance.
Model-Based Techniques Model-based methods can further improve sample efficiency, such as MBPO (Janner al. 2019) using synthetic rollouts and Dreamer (Hafner al., (023) showing strong data efficiency in real-world tasks (Wu et al.) 2023). However, they often still require stochas- tic exploration in the environment. (unn et al. (2025) train a world model that explicitly reconstructs the environment state and use it as policy input to enhance locomotion robustness, but the model is not used to generate synthetic data for policy fine-tuning aimed at improving adaptability. ASAP (He) et al.) 2025) collects real- world data using pretrained policy and learns a delta-action model to cor- rect simulator actions before fine-tuning. However, the delta network can output unbounded action corrections while finetuning, which may lead to undesired behaviors, and the approach requires substantial real-world data to sufficiently cover the gaps between the simulator and the physical robot, especially across diverse motions. These limitations highlight an open question: how can physics priors be incorporated into world models to improve predictive accuracy while reducing the amount of data required for policy fine-tuning? Physics-Informed Neural Networks (PINNs) (Raissi) et al. 2019) represent one possible approach by embedding physical constraints directly into the learning architecture. For example, Greydanus al. (2019) proposed Hamiltonian neural networks that learn energy-conserving dynamics in an unsupervised manner, while Cranmer et al. (2020) introduced Lagrangian neural networks that model dynamics without requiring canonical coordi- nates. Deep Lagrangian Networks (Lutter et al. 2019) further demonstrated strong performance in robot tracking and extrapolation to novel trajectories. Yet these approaches remain limited to energy-conserving systems and struggle with contact-rich dynamics. To address this, SSRI (Levy) jet al. 2024) proposed combining rigid-body dynamics with learned contact-force models, enabling quadruped robot to learn to walk with just three minutes of real-world data. Nevertheless, training from scratch remains unsafe for humanoids due to their fragility and instability. Motivated by this, pipeline that extends physics-informet modeling to large-scale
our work adopts a pretrain- humanoid pretraining and safe finetuning.
3 PRELIMINARIES
Problem Formulation. RL is typically modeled as a Markov Decision Process (MDP) (Altman) 1999]: M = AP R, n, 7), where S and A denote the state and action spaces, respectively. P(s'Ts, [s, a) and a) represent the transition probability and reward function. μu()) denotes the initial state distribution, and 7 is the discount factor. A stochastic policy тo parameterized by 0 specifics the action probability Ta(als) for a given state s. In continuous control, the stochastic policy is typically Gaussian, Th (als)) = (s)(s,,, Eo)s),, where the covariance can be either state- independent or state-dependent. During pretraining, domain randomization is employed to enhance robustness by perturbing the environment transition function P across episodes. At the same time, the stochastic policy is used to promote diverse exploration. For the deployment and finetuning stage, however, we adopt the deterministic policy induced by the mean action pp(s) s) to collect data and evaluate performance in environments.
3

---

Published as a conference paper at ICLR 2026
Model-based RL Problem. the model-predicted return Jo (Te), defined as follows:
For the finetuning stage, we seek policy my Hy that maximizes
mopp ae To(-1s)), (8141,7011) ~ Po(. se st
(TO) =
rret1
(I)
Here (- ae) denotes a world model parameterized by é, which approximates the true environ- ment dynamics. The initial state 50 sampled from the true initial distribution μe, while subsequent states are generated by the learned model. In practice, stochastic policy To is used to interact with Fs so as to generate diverse imaginary trajectories, thereby improving exploration coverage for pol- icy optimization. When the predictions of the world model are sufficiently accurate, maximizing the model--preiccte return corresponds maximizing the return in the real environment (Janner al, 2019 Hafner et al. 2023).
Physics-Informed World Model for Robots Incorporating physics priors into the world model improves the accuracy of its predictions (Raissi et al.) 2019). We adopt physics-informe formu- lation of robot dynamics based on the Lagrangian equations of motion (Featherstone 2008):
( + Clge. d)) + G(q.) = BT, B +
Here, qi and qt are the generalized coordinates and velocities at time t. The motor torque Ti is applied to the joints via the matrix B. FE is the external contact force with Jacobian J, and 7 denotes dissipative (parasitic) torques. The matrices M, C. and G' are the mass, Coriolis/centrifuaall and gravity terms. Because B, M, C, and G depend only on the robot's s shape and inertial parameters, they are known beforehand. Thus, the main model uncertainties are the contact term J F? and the dissipative term T/" To model these unknown contributions, Levy et al. (0024) introduce residual network that approximates them. The network predicts the sum of the contact term and the dissipative term, as well as the uncertainty of the next-state prediction:
To(s;, (+) ~ of = P +
(3)
(o(5; a:) log of
(4)
Here, To and 0 are outputs from different heads of the neural network, with of representing the predicted variance of the next state. This hybrid physics-informed world model P combines the known rigid-body dynamics with learned residuals and can be rolled out to generate more precise trajectories for policy optimization.
LARGE-SCALE PRETRAINING AND EFFICIENT FINE-TUNING
To enable humanoid robot to learn quickly and with high sample efficiency in new environments, we first select the algorithm best suited for fine-tuning and then design the framework around it. Using a single algorithm across both stages aligns training objectives, enables continuous policy updates that mitigate forgetting. and provides unified, cusy-to-implement framework. While SSRL (Levy et al.) 2024) successfully trained a quadruped robot from scratch using SAC and a physics- informed world model, its reliance deterministic data collection leads to impractically slow run-times when scaled to humanoid robots. Combining this approach with PPO is a natural alternative, PPO offers extensive parallel training infrastructure (Schwarke et al. 2025). How- ever, our preliminary study (see Appendix II highlights two advantages of SAC over PPO: its off off-policy nature ensures stable convergence with limited samples, and its state-dependent stochas- tic actor promotes exploration in the world model, enhancing the diversity and utility of synthetic rollouts. Nevertheless, in our setting, none of these methods complete the target task within 48 hours of runtime without large-scale parallel pretraining, motivating the development of full pre- training-finer fineting pipeline using SAC as the backbone algorithm.
4.1 LARGE-SCALE POLICY PRETRAINING
Policy Optimization. We adopt Soft Actor-Critic (SAC) (Haarnoja et al. 2018b) with an asym- metric actor- critic setup (Pinto et al., 2017): the actor na (a(a, receives state s,, while the crities Q (sf, at) are trained privileged states of that include additional information. method uses
4

---

Published as a conference paper at ICLR 2026
only the robot's proprioceptive state for both the actor and the critics. We normalize each feature of St and sy using rumning mean and variance (Lee et al. 2024). ensuring balanced feature scales dur- ing training. We pretrain the SAC policy in MuJoCo Pla Playground (Zakka et al. 2025) with thousands of vectorized environments on a single GPU. Our SAC implementation is written in JAX (Bradbury)
ee al. 2018) with fixed tensor shapes, which allows efficient operation fusion and reuse of compiled kernels. As a result, large-batch updates (high UTD) incur no additional data transfer overhead. The same design seamlessly accelerates interaction with the world model during finetuning and sup- ports multi-GPU scaling, improving wall-clock training time. Unlike FastTD3 (Seo et al. (225 and PQL eLi | [003)), we do not use per-environment mixed Gaussian noise. Instead, we rely entirely on SAC's stochastic policy for exploration, i.e., aa 8) = (Nu (+), (s)), with state-dependent variance produced by the actor and temperature a controlling entropy. We employ the Optuna framework (Akiba et al., 2019) for systematic hyperparameter tuning. On the Booster TI locomo- tion task, we conducted approximately ten hours of hyperparameter search on a single NVIDIA RTX 4090 GPU. The results show that, after tuning, the convergence time is reduced from about seven hours to only half an hour. The training objective and hyperparameter tuning process are detailed in Appendix B.I
Large-Batch Updates and High UTD. Raising the update-to-data (UTD) ratio reuses experi- ence more aggressively but can amplify value-estimation bias. In our T1 TiLowDimJoyst inJoysti setup- SAC trained in MuJoCo P Playground with 1,024 parallel environments, large-batch up- dates (batch size 1,024), and a 100-transition replay buffer-we observe that increasing UTD from to 10 improves sample efficiency without any auxiliary stabilizers or architectural modifications. Be- yond this range, gains diminish while wall-clock time increases predictably. We do not claim gener- ality: these improvements appear tied to massively parallel, domain- likely suppresses early overfitting and curbs bias accumulation at moderate UTDs. By contrast, prior works typically couple high UTD with stabilizers such as periodic resets (D'Oro et al. 2023 Nikishin et al., (022) or architectural regularization (ensemble/dropoutt critics, natch-normmalized critics) (Chen al.] [202]] Hiraoka et al., 2022; Bhatt al., (024). We view these techniques as complementary and worth re-evaluating in large-scale parallel training for SAC to potentially push the usable UTD higher.
regime, which
4.2 WORLD MODEL PRETRAINING
Offline Pretraining from Logged Transitions Unlike MBPO or Dreamer, which update the pol- icy and world model together online, we decouple them to improve wall-clock efficiency under massive parallelism. With thousands of parallel simulators, each step generates on the order of O(19)) transitions, and training the world model online would slow the training. During SAC pre- training. we log all transitions to disk, saving records of the on-device replay buffer. After the policy converges, we train the world model entirely offline from these records. From each record, we con- struet D = (sf, at, sp s+AA)- In our setting, the privileged state is sp = [gt, de: vp, wh. ne, he]. which includes joint angles and velocities, base linear and angular velocity in the body frame, base orientation, and body height.
World Model and Loss. We learn residual predictor that injects external torques into differ- entiable physics step. Residual predictor takes the concatenation of the privileged observation and action [s], ai] as input and outputs external uncertainties of torques and per-feature predictive un- certainty of the next-observation prediction:(7f. log az) shown in Eq. (3) Given [s] as, 7f we predict the next privileged state using fully differentiable pipeline. Concretely, we implement the world model based on Brax (Freeman et al., (02T)) and perform the following steps: (a) map privi- leged state to Brax generalized coordinates and velocities, (b) convert actions to motor torques via a PD controller, (c) integrate the Lagrangian dynamics with semi-implicit Fuler, and (d) reconstruct the next privileged state for learning and rollout. Using Brax's differentiable rigid-body primitives
M (q). C(q. q), and G(q)) - we avoid reimplementing the dynamics and keep the rollout loss pipeline end-to-end differentiable. See Appendix B.2 for details.
Compared to SSRy(Levy et al. 2024). we (i) correct the mapping from privileged state to Brax's generalized state (g. 9), resolving a discrepancy identified in the publicly available SSRL imple- mentation; and (ii) align each observation/state dimension between MuJoCo Playground and Brax (frames, quaternion form, normalization) to avoid training error we include the base height h
5

---

Published as a conference paper at ICLR 2026
in the privileged state. For humanoid tasks, omitting hy consistently caused unstable world model rollouts and non-convergent fine-tuning, whereas SSRL's quadruped configuration omitted height without issue. These adjustments improve training stability in humanoids setting.
After obtaining the next privileged state S7+ from the differentiable pipeline, we minimize the negative log-likelihood of Gaussian predictive distribution. Let B denote the batch size. For prediction & t+ target s 14 and elementwise log-variance log of. the loss is
C
(B - ① exp( - log oo, + log ),,)) B b=1
(5)
where @ elementwise multiplication. Gradients backpropagate through normalization, frame transforms, the PD controller, and the Euler Step, the residual predictor is trained end-to-end from next-state errors. Minimizing this loss encourages the world model to predict small variance for seen states and large variance for unseen states. In practice, we exploit this behavior during rollouts by sampling next states from the predicted Gaussian S $/4 ~ ~N diag(o?)).
4.3 POLICY AND WORLD MODEL FINETUNING
Deployment & Data Collection. We deploy the pretrained policy in the target environment (Brax (Freeman et al. 2021) in our case) and collect trajectories using deterministic actions (i.e., the action mean, without exploration noise). If the robot enters an unsafe state, the episode reset. Each episode has maximum length of Top 1000. All transitions are stored in replay buffer, which is subsequently used to fine-tune both the world model and the policy. After collecting Tep steps of data, fine-tune the world model and policy before beginning the next iteration of data collection and training. More sophisticated pipelines, such as asynchronous data collection and learning (Luo et al. 2024), are left for future work avoid additional complexity in this study.
World-Model Fine- Tuning. We tune the pretrained world model on the collected buffer. For each epoch we independently shuffle data, roll physics inside the loss, and evaluate world model on a held-out testing data set for early stopping. We observe that training world model for multiple epochs on the same replay buffer, combined with auto-regressive training, enhances sample efficiency. The auto-regressive training objective is provided in the Appendix B.3 After World model training's s done, we use the policy to explore in the world model to generate data that used to train the actor critic. Once world-model training is complete, we use the stochastic SAC actor to explore within the world model and generate trajectories.
Exploration in Physics-informed World-Model. (a) Asymmetric Observation Generation. Be- cause data are collected with a deterministic policy (action mean only), the logged trajectories have limited diversity. To drive policy improvement during finetuning, we therefore explore inside the world model using a stochastic SAC policy. We keep the same asymmetric structure for actor-critic during finetuning as in pretraining: the actor consumes the proprioceptive state, while the critic uses the privileged state. Let s E &d denote the privileged state and s; E IR the proprioceptive state, from which the action a₂ ~ TT(-(s)) sampled. The learned physics-informen world model Pa predicts the next privileged state, 8,41 by fixed, simulator-consisten projection II + Rd
Poo az)) and the proprioceptive state is obtained
Rdd (same deterministic transforms used (1)+))). (b) Safety Reset. For safety and numerical robustness, apply physies-informed terminal checks and terminate a world model rollout immediately upon detecting an abnormal state-specificallyy when base height, world-frame linear/angulat velocities, body roll/pitch, or joint positions/velocitie violate prescribed bounds. Early termination stabilizes training and prevents NaNs over long horizons. We leave richer safeguards- e.g.. self-collision checks derived from link poses-aa as future optimization. (c) Physics-Consistenn Reward. Since 3 provides all necessary quantities (joint states, link poses, base kinematics), rewards can be com- puted deterministically and remain fully consistent with the underlying physics in the world model rollout. An example is provided in the Appendix B.4 In contrast, using learned scalar predictor (o ..) (Hafner et al. (022)) is unstable in deterministic data collection settings. Even small state prediction emrors accumulate over long horizons. degrading policy convergence. We sample batch of privileged states from the replay buffer as initial conditions and roll out trajectories of horizon Hom 20 using the world model and the stochastic policy. These synthetic trajectories are added
Rd.
by the environment): 5.+1

---

Published as a conference paper at ICLR 2026
to the replay buffer train the actor-critic via SAC. The updated policy is then deployed in the environment to collect new data, thus completing one cycle of an iterative process that continues until convergence.
EXPERIMENTS
Our approach performs comparably to baselines in pretraining and, importantly, enables efficient fine-tuning for humanoids in new environments with limited data. We validate on two plat- forms-Booster TI and Unitree Gl-and report LIFT results averaged over independent runs. All experiments are executed
single NVIDIA 4090 GPU with 32 CPU cores in cloud set-
ting. Evaluation uses the average undiscounted return over E episodes, ( T Σ Ex Pts computed without network updates. In practice, we set E = 1024 and Tep 1000. To characterize efficiency end-to-end, we compare (i) wall-clock time during pretraining (training speed) and (ii) en- vironment steps required during fine-tuning (sample efficiency). These experiments are designed to answer three key questions: (Q1) Is the LIFT pretraining module comparable to established base- lines? (Q2) Does this module better support subsequent tine-tuning than baselines? (Q3) Which components of our pipeline affect fine-tuning efficiency and stability?
(Model-free): An efficient variant of TD3 with parallel environments and mixed exploration noise. (2). PPO (Schul-)
Baseline algorithms. The baselines include: (1). FastTD3 (Seo et al.
2025)
man et al.) (017)) (Model-free): A widely used on-policy algorithm with clipped surrogate objective. We use baseline PPO in Mujoco playground. (3). SAC (Haarnoja et al. (0188) (Model-free). An off-policy algorithm with stochastic actor and entropy regularization to encourage exploration. (4). SSRL (Levy et al. 2024) (Model-based). A physics-informed world model approach for training policies from scratch. (5). MBPO (Janner et al. 2019) MMdee--baedd:: Utilizes an ensemble of neural networks as the world model to improve sample efficiency of RI..
5.1 PRETRAINING EXPERIMENTS
To answer Q1, policy pretraining is performed in the MuJoCo Playground (Zakka et al. 2025) across six humanoid tasks, combining two terrain types (flat and rough) with three robot configurations: a low-dimensional Booster T1 (12-DoF, legs only), full Booster T1 (23-DoF, including head, waist, arms, and legs), and the Unitree G1 (29-DoF, including waist, arms, and legs). Full details are provided in the Appendix C.2
Pretraining results. As shown in the Appendix A.2 LIFT achieves comparable or higher evalu- ation returns than PPO and FastTD3 while stabilizes at its peak return faster rough terrain envi- ronments. On flat terrain, achieves comparable peak performance with similar wall-clock runtime. These curves indicate that the LIFT pretraining module competitive in reward returns and conver- gence speed. Additionally, we also apply LIFT pretraining to whole-body tracking tasks for Unitree GI, providing preliminary evidence that the same framework extends naturally non-locomotion skills. More details and discussion of these results are provided in Appendix A.8
Sim-to-real with LIFT Pretrained Policy. To demonstrate the potential for zero-shot deploy- ment, we deploy the pretrained policy for the low-dimensional Booster TI directly on the physical robot. Zero-shot transfer to previously unseen surfaces (c.g., grass, uphill, downhill, mud, etc.) is shown in the Appendix A.6] This provides practical evidence that large-scale, parallel SAC pre- training can yield deployable humanoid controllers, thereby addressing Q1. And it also establishes suitable starting point for subsequent fine-tuning.
5.2 FINETUNING EXPERIMENTS
Sim-to-sim Finetuning. We evaluate finetuning in Brax (Freeman et al.) 2021) after large-scale pol- icy pretraining the MuJoCo Playground. The world model is also pretrained with data collected with pretrained policies. Because the two simulators differ in contact models and constraints, this sim-to-sim transfer provides a controlled but nontrivial test of adaptation. We transfer the pretrained policy to Brax and keep the reward design same as Booster Gym Wang et al. (2025) for both pre- training and finetuning. During pretraining, target linear velocities along the 7-axis were uniformly [-1, 1] m/s. For fine-tuning in Brax, specify new forward-velocity targets and the sampled in

---

Published as a conference paper at ICLR 2026
policy explores with deterministic action execution (action mean only; stochastic sampling in the environment). We measure (i) whether policies converge within a fixed sample budget and (ii) velocity- -trccitgtrackig accuracy relative to the specified target. Implementation details are provided in the Appendix
C,3
(In-Distribution)
(Long-Tur Taily
m/s (Outaaf-Disributtion) mis (Ons-offDistribuiionn)
Eval Episede Return
v0- 60-
50
60-
Forward Lincar Nelecilys-acs
0.5
0.55
D..-
40
5.0
32 Environment Stepot (10")
24
0
24 32 40
Environment Stepo(H (10")
(10p))+107)
Environment Ste
LIFT (Ouns)
SAC
PPO
FasIDD3
SSRL
Figure 2: Results of finetuning Booster T1 robot with varying target speeds. The black dashed line represents the target velocity for each task. Results are averaged over 8 random seeds.
To address Q2, we design three scenarios and compare against PPO- and BastTDD3prettringg poli- cies: Dittiiisttibuttinn: targets within the pretraining range; (ii) Long-Tail: rare targets poorly represented during pretraining: (iii) iittrf-Distribbttin: targets outside [-1, m/s.
As shown in the Figure LIFT consistently converges across all tasks, achieving stable walking that closely tracks the desired forward speed during the Booster T1 finetuning experiments. After fine- tuning, the policy demonstrates significantly reduced body oscillations and less deviation from the desired speed direction, with noticeable improvements in velocity. In contrast, SAC, trained without explicit exploration noise, quickly diverges and fails recover, indicating rapid overfitting to the deterministically collected data. Although PPO's clipping mechanism stabilizes updates by keeping the new policy close to the previous one, performance in our setting initially remains reasonable but then gradually degrades and ultimately collapses. This suggests that, under deterministic exe- cution and limited data collection, PPO struggles sustain stable policy improvement. FastTD3 exhibits strong oscillations and ultimately collapses without converging. SSRL shows signs of con- vergence at 0.6 m/s but fails to reach the target speed, and does not converge at all on higher-speed tracking tasks (1.0-1.5 m/s). This highlights the difficulty of training humanoid locomotion from scratch and further motivates our pretrain-finctume finetune design. These results demonstrate that LIFT enables robust adaptation under both in-distribution and out-of-distribution finetuning tasks, while standard baselines struggle with stability. During finetuning, policy control and data collection run 50 Hz (one environment step 0.02 s), so 4 x 10 steps correspond to approximately 800 s of on-robot interaction, highlighting the potential feasibility of applying LIFT on real humanoid robots which remains future works. Additionally, we finetuned the Unitree GI in the Brax environment, which also improved policy behavior and reward performance. The detailed results can be found in the Appendix A.7
Real-world Finetuning, We evaluate LIFT on the Booster TI humanoid by first pretraining a policy in the MuJoCo Playgrood T' LoLLooomoooyytiie/llaa Flateerrin task with most energy-related regulariza- tions removed, keeping only an action-rate 1.2 penalty. This policy transfers well from MuJoCo to Brax but fails in zero-shot sim-to-real, providing challenging starting point for real-world finetun- ing. Using this policy as initialization, our finetuning framework progressively improves initial un- stable behavior: after collecting 80-590 s of data, the robot shows more upright posture, smoother gait patterns, and more stable forward velocity (Fig. B). demonstrating that LIFT can substantially strengthen a weak sim-to-real policy with only several minutes of real data. However, our current real-world implementation has two main practical limitations: (1) LIFT requires base-height
8

---

Published as a conference paper at ICLR 2026
timation for world-model training and rewards, but Booster TI does not provide this onboard, so we rely on Vicon motion-capture system, which restricts the tracking area and requires human supervision. (2) We estimate base linear velocity by integrating IMU acceleration, which introduces drift and may limit policy tracking performance. (3) Each finetuning iteration is executed sequen- of data collection at 50 Hz, world model update, then synthetic rollouts for policy updates-Icading to multi-hour wall-clock time and frequent battery swaps despite using only min- utes of real data. We view these engineering rather than conceptual constraints, and expect that adopting an asynchronous pipeline similar to SERL (Luo et al. 2024). together with camera- based height and velocity estimation onboard, would make repeated real-world finetuning substantially more practical. More results are provided in Appendix[A.3
ui
collecting data
collecting collecting collecting collasting
Figure 3: Real-world finetuning progression on the Booster Tl humanoid. video demonstration is available on our project website.
5.3 ABLATION STUDY
To answer Q3 and identify which components of LIFT affect fine-tuning efficiency and stability, ablations over three aspects are conducted: (i) pretraining stages-large--e SAC pretraining and world-model (WM) pretraining: (ii) world-model choice our physics-informed model versus an MBPO-style ensemble; and (iii) key hyperparameters (pretraining), entropy coefficient a and autoregressive loss horizon (fine-tuning).
ratio, replay-buffer size, batch size
Effect of Pretraining. As shown in the Fig- ure 4 we ablate WM and SAC pretraining.
Ti - 1.Stic velucity
Ti 1.5mmss velucity
Fooand Velechyf--dt
With both pretraining stages, LIFT converges within x 10s environment steps and success- fully tracks the target speed. Removing WM pretraining still allows the method to converge eventually, but noticeably slows down train- ing, indicating that WM pretraining improves sample efficiency. Removing both pretraining stages reduces the LIFT to SSRL Levy et al. (2024), which mostly learns stand in place with near-zero forward velocity. Thus, large- scale SAC pretraining essential avoid poor local minima, and WM pretraining further im- proves finetuning efficiency and stability.
Eval Ejtoade
W
84.
32 Eavironment x10P
Emirenment Steps(=10")
LIFT (Cuns
LIFT wilhout WM Pestraining
SSRL (without WM and SAC Pretraining!
Figure 4:
Ablation of the pretraining on Booster T1 (target forward speed 1/5). Re- sults are averaged over 8 random seeds.
Physics-informed vs. Non-physics-informe world model (ensemble size 5, elite size = 3) on the same dataset and finetune with identical hyperparameters to LIFT; the only difference is the choice of world model. MBPO fails to con- verge: episode return remain near zero (training curves as shown in Figure 5). On the test set its mean squared error (MSE) of world model is substantially worse than LIFT's. s. During model-based rollouts, a stochastic policy frequently produces actions that outside the distribution that has been seen in world-mode training. These out -ut-of-disttibution actions induce physically implausible predictions (c.g., body height) for MBPO, which cause the critic loss to explode and inhibit pol- icy improvement. This behavior likely stems from the purely neural network's limited ability to
World Models We pretrain MBPO's ensemble

---

Published as a conference paper at ICLR 2026
generalize. By contrast, LIFT's physics- innfoee- world model supplies strong inductive priors that improve generalization under limited data and produce stable, learnable rollouts, enabling successful finetuning.
velccity
T1- Smils velccity
Smiis velocity
valocit
MBPO
Rval Epoonde Retum
Tol MSE Lost
Height
LIFT (Cus))
00-
0000
01011
0:72
15
01003
16
24
8.30
Envinorment Stapst 10"))
Environment (100)
Environment Steput
Enctronment Stepst -100)
Figure 5: Ablation of Physics informed World Model on Booster Tl (target speed = 1.5 m/s). Results are averaged over 8 random seeds.
Effect of Hyperparameters. During pretraining, we find that the UTD ratio, buffer size, and batch size strongly influence convergence speed and performance. UTD = 1 converges very slowly, while increasing to speeds up learning. Larger buffer and batch sizes further accelerate conver- gence, though excessively large buffers increase GPU memory usage. Beyond certain thresholds, larger batch sizes or higher UTD provide little additional benefit. In finetuning, the entropy coef- ficient and autoregressive horizon length are critical. Excessively large promotes exploration into high-un sigh-uncertainty regions of the world model, destabilizing learning. Using a from pretraining works reasonably, but smaller values achieve more stable convergence. A loss horizon length of sometimes fails to reach the target velocity, while lengths of or consistently ensure stable learn- ing, indicating that multi-step autoregressive prediction improves world-model training and policy stability. The training curves are provided in the Appendix A.4
6 CONCLUSION AND DISCUSSION
In this work, we propose LIFT, a pretraining-fnnttninng framework for humanoid control that bridges large-scale simulation and data-efficient adaptation. By leveraging massively parallel en- vironments with SAC, LIFT enables fast, robust pretraining and zero-shot sim-to-real transfer. The pretrained policy then guides model-based fine-tuning, while the same pretraining data bootstrap a physics-nffrmedd world model to improve sample efficiency. During fine-tuning, deterministic action execution is combined with stochastic exploration inside the world model, providing stable learning under limited data. Experimental results highlight a practical path toward continuous, effi- cient humanoid learning.
Despite these results, there still exist limitations in our current work that inspire several future di- rections: (1) Safety management during real-world finetuning: collecting deterministic trajectories in the real world still carries inherent risks due to model errors in the actor network. We enforce strict termination conditions, aligned with those in simulation, to halt episodes when key physical quantities exceed predefined thresholds. Human operators also remotely terminate any episode ex- hibiting unsafe behaviors (e.g., drifting toward obstacles). Upon termination, the policy is halted and the Booster T1 is switched to damping mode to restrict further motion. The robot is then guided back to the starting area using its default walking controller, followed by a standing con- troller reset it to a consistent initial configuration. This procedure maintains stable initial states across episodes and mitigates distribution mismatch during data collection. Future work may in- corporate more automated safety mechanisms -ucinnsss-chhhh as robot-assisted resetting (Hu et al. (025). uncertainty-awarr exploration (An et al.) 2021: Yu et al. 2020). or recovery policies with safety switches (Thananeyyan et al., 2021| [Smith et al., 2022a to further improve safety and reduce hu- man intervention during real-world humanoid finetuning; (2) high-dimensional external sensor and visual inputs: LIFT currently operates exclusively on proprioceptive observations and does not in- corporate camera or other high-dimensional sensory inputs, in contrast to vision-based frameworks such as DreamerV3 (Hafner et al. 2023).. Scaling LIFT to tasks with rision-centris objectives such as dexterous manipulation or object-centre control will likely require latent world models capable of capturing dynamics beyond the robot's body state, which we view as a promising direction for future extensions.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China (No. 62403064, 62403063) and Shenzhen Science and Technology Program (No. ZDCY2025000099431000) We thank the engineering team from Booster Robotics for technical support.
Reproducibility Statement. We train all LIFT agents on a single NVIDIA RTX 4090 GPU. Our experiments the MuJoCo Playground and Bray simulators. The source code and full results are available on our project website: httpp:: /tt:://lii /iift wwrtt-hhaannniin implementations are adapted from the following open-source repositories: PPC FastTD: MBPO, and SSRI Detailed hyperparameters, additional experiments, and environment configurations are provided in Appendix A and
io/ / Our baseline
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-gentt next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pp. 2623-2631, 2019.
Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline inforcement learning with diversified q-ensemble. Advances in neural information processing systems, 7447, 2021.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In International Conference on Learning Representations (ICLR), 2024.
URL /tttps://opereeeeeee net.
net forum? 1oram21dd-ccQQtTTTII
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Minn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github.com wotp///mm//aaaa/a/axaaa
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-
learning: Learning fast without model. In International Conference Learning Representa-
tions (ICLR), 2021. URL httphtttts//opeeeeeeee
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint 2xXiv:20030046.33 2020.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G. Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio bar- rier. In International Conference on Learning Representations (ICLR), 2023. URL https: //operreview. ieeerrawweee.nettoooommiiaaieaee http:/foorummmd00CC-p-- 9aBBVJe
Roy Featherstone. Rigid body dynamics algorithms. Springer, 2008
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint //0.:2//061/288.. 2021.

---

Published as a conference paper at ICLR 2026
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation enror in actor-critic methods. CoRR, abs/1802.09477,, 2018.
09477
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019.
Xinyang Gu, Yen-Jen Wang, and Jianyu Chen. Humanoid-gym: Reinforcement learning for hu- manoid robot with zero-shot sim2real transfer. arXiv preprint arXiv:2404.056995 2024.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint /803. 2(3), 2018.
Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort. arXiv preprint arXiv:2002.08550 2020.
Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. arXiv preprint arXiv: 1812.110011113 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. Pmlr, 2018b.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05900 2018c.
Danijar Hafner, Jurgis Pasukonis. Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint 2.Xi:23301.0400. 2023.
Nicklas Hansen, Jyothir SV, Vlad Sobal, Yann LeCun, Xiaolong Wang, and Hao Su. Hierarchical world models visual whole-body humanoid controllers. arXiv preprint arXiv:2405. 18418, 2024.
Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbab, Chaoyi Pan, et al. Asap: Aligning simulation and world physics for learning agile humanoid whole-body skills. arXiv preprint arXiv:2502.01143, 2025.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka.
Dropout q-functions for doubly efficient reinforcement learning. In International Conference on Learning Representations (ICLR), 2022. URL. https://p/// http///ppeeevveeeeeeeeee ew net/ / forum?ic
3CVJMMPV3RT.
Kaizhe Hu, Haochen Shi, Yao He, Weizhuo Wang, C Karen Liu, and Shuran Song. Robot trains robot: Automatic real-world policy adaptation and learning for humanoids. arXiv preprint arXiv:2508. /2252, 2025.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in neural information processing systems, 32, 2019.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jacgul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint 2.X0::2410009744. 2024.
Jacob Levy, Tyler Westenbroek, and David Fridovich-Kan Keil. Learning to walk from three minutes of real-world data with semi-structture dynamics models. arXiv preprint Xiv:24100009166 2024.
Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel g-learning: Scaling off-policy reinforcement learning under massively parallel simulation. In International Conference on Machine Learning, pp. 19440-19455 PMLR, 2023.
Qiayuan Liao, Takara E Truong, Xiaoyu Huang. Guy Tevet. Koushil Sreenath, and C Karen Liu. Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion. arXiv preprint arXiv:2508.082. 2025.
12

---

Published as a conference paper at ICLR 2026
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint WXt:::5509009900 2015.
Jianlan Luo, Zheyuan Hu, Charles Xu, You Liang Tan, Jacob Berg, Archit Sharma, Stefan Schaal, Chelsea Finn, Abhishek Gupta, and Sergey Levine. Serl: A software suite for sample-efficient robotic reinforcement learning. In 2024 IEEE International Conference on Robotics and Automa- fion (ICRA), pp- 16961-16969. IEFF, 2024.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. arXiv preprint arXiv: 044400449. 2019.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 108. 10470, 2021.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In Proceedings of the 39th International Con- ference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Re- search, pp. 16828-16847) PMLR, 2022. URL
www.hnn.........mmmm.mm.......m. press/
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. arXiv preprint, arXiv:1710.06542, 2017. URL
httpos//arxxv.oorr//a////11 06542
Antonin Raffin. Getting to work massive parallel simulator: An rl journey with off oof-cc algorithms. craffin.ithubb.. Feb 2025. URL https::/ github. wtcii//psss iotost//
sac massive sim/
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pp. 91-100. PMLR, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.00777 2017.
Clemens Schwarke, Mayank Mittal, Nikita Rudin, David Hoeller, and Marco Hutter. Rsl-rl: A learning library for robotics research. arXiv preprint arXiv:2509.10771 2025.
Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Chao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint arXiv:2505. 22642, 2025.
Laura Smith, J Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In 2022 international conference on robolics and automation (ICRA), pp. 1593-1599. IEEE, 2022a.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv: 07860, 2022b.
Wandong Sun, Long Chen, Yongbo Su, Baoshi Cao, Yang Liu, and Zongwu Xie. Learning humanoid locomotion with world model reconstruction. arXiv preprint 2002.22502. 2025.
Brijen Thanamjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, (3):995--4922, 2021.
13

---

Published as a conference paper at ICLR 2026
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Yushi Wang, Penghui Chen, Xinyu Han, Feng Wu, and Mingguo Zhao. Booster gym: An end-to-end reinforcement learning framework for humanoid robot locomotion. arXiv preprint arXiv:2506.15132, 2025.
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. Conference on robot learning, pp. 2226-2240.
PMLR, 2023.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14442 2020.
Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground. arXiv preprint arXiv:2502 088444 2025.
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTS
A.1 PRELIMINARY STUDY
We consider the practical case where the pretrained policy entirely fails in the deployment stage, finetuning must effectively learn new policy from scratch, with data collection constrained to de- terministic policy. To study this setting, we design a simple experiment using the BoosterTi robot, which has 12 lower-body degrees of freedom. The reward is composed of three terms commonly used in humanoid locomotion: a swing-leg reference, body-height tracking, and linear velocity tracking. The policy is trained to achieve target forward velocity of 0.2 m/s. m/2m/s. The implementation
details can found in Appendix C.I
As shown in Figure 6 we compare PPO and SAC within the SSRL (Levy al. 2024 framework. The default setting uses SAC, which leverages a state-dependent stochastic actor to explore in the world model and improve policy performance. For comparison,
replace SAC with PPO (imple- mented in RSL-RL (Schwarke et al., (025)) while keeping all other components fixed, tuning only the hyperparameters. We observe that the action standard deviation (std), which strongly influences the distribution of states explored in the world model. A larger makes training difficult converge and tends drive the policy into regions where the world model is poorly trained, while a smaller std leads to under-exploration and unstable convergence, with some seeds failing to learn at all. To mitigate this, we replace the PPO actor with the SAC actor that outputs both mean and std of the action distribution. This modification greatly improves stability, with all seeds converging. but still requires two to three times more samples than SAC. Meanwhile, jet al. 202T) simulation. None of the methods converge to the target velocity within this time, high- lighting the need for large-scale pretraining reduce wall-clock time and avoid convergence to local minima.
critical factor for PPO performance
the initial
run these experiments for 48 hours on single Brax (Freeman
Preliminary Study
Preliminary Study
Forward Linear Velecity(x-axiis)
0.15
Eval Eppisode Retur
20-
0.10
0.00
500
100
150
200
500
100
150
200
Environment Steps(x103)
Environment
Steps(x103)
SSRL-PPO-L.0 std
SSRL-PPO-0.7 std
SSRL-PPO-0.4 std
SSRL-PPO-std head
SSRL
Figure 6: Comparison of PPO and SAC a preliminary fine-tuning experiment with the BoosterTi robot, run for 48 hours on single Brax simulation. The left figure shows the evaluation episode return, and the right figure shows the body's linear velocity along the z-axis. Curves represent the average over 8 random seeds, with evaluation performed in 128 parallel environments. PPO performance is highly sensitive to the initial action standard deviation (std). Replacing the PPO actor with SAC-style actor that outputs both the mean and standard deviation improves stability, although SAC still requires fewer samples to achieve the same performance.
15

---

Published as a conference paper at ICLR 2026
A.2
PRETRAINING EXPERIMENTS
TlLowDDinooyyttiRRuuggh. Temmin
T moyys
GileessiikRoughheema
10
Epioods
TUJoysie&Fa/Teeriin
GlJoystickFaaTTeraia
120
Epicode
60
30
Runtime (5) (x10F)
Runtine (5) (x10F)
Runtime (6) (x102)
LIFT(Ours)
PPO
Fas:TD3
Figure 7:
Pretraining performance comparison of LIFT (red), PPO (orange), and FastTD3 (blue) across six humanoid tasks (top row: rough terrain for the three robot configurations: bottom row: flat terrain for the three robot configurations). Results show the mean over random seeds.
A.3 FINETUNING EXPERIMENTS THE REAL- WORLD
T1 0.6 m/s (Real World)
TI 0.6 m/s (Real World)
Forward Linear Velocity(X-aNe)) (x-aiss)
0.60
mmm
0.48
Episode Return
24-
0.36
16
0.24
8-
0.12
Reward of Tracking Angular Velocity
0.00
0.75
Reward of Action Rate
0.60
-0.06
-0.12
0.30
-0.18
0.15
-0.24
LIFT (Ours)
10
15
20
25
10
15
20
25
Environment Steps(x103)
Environment Steps(x109)
Figure 8: Finetuning performance of LIFT in the Real-World. Across random seeds, LIFT con- sistently improves episode return, forward velocity tracking. and angular-velocity tracking, while reducing the action-rate penalty. Note that the velocity tracking eπтог (target at 0.6 m/s) might come from the noisy base-velocity estimation from the onboard IMU accelerator.
16

---

Published as a conference paper at ICLR 2026
A.4
THE EFFECT OF HYPERPARAMETER
T
120
Eval Epioade Return
90
UTD
Batch Sizu
256
30
Buller Size
16906
312
1024
100300
4095
1000002
5152
"30"
Runtime (5) (x10")
Runtimee (6) (x10")
Runtime (5) (x10")
Figure 9:
Effect of pretraining hyperparameters on Low dimensional Booster T1. From left to right: UTD ratio, buffer size, and batch size. UTD values compared: 1, 5, 10, 15, 20; buffer sizes: 10,000, 100,000, 1, 1,000,000; batch sizes: 256, 512, 1,024, 4,096, 8,192. Larger UTD and batch sizes accelerate convergence; increasing buffer size improves learning speed but at the cost of much higher GPU memory usage.
Ti 5mik velecity
Focead Hneu (elecity--aaa))
T1 Smt's velucity
T1 11-1, velucity
Furwerd Lincar Milucity socb
Ti 1.5mis velecity
60-
Environment Steps(=10")
Envtronment Steps/ 10")
Enstronment
Stepel -10")
Environment Stepe(=10")
0.001
LIFT
LIFT Los Hocizon
LIFT Loss Horizon
LIFT (orr)Onns)-aaae alppa 00.1
LIFT (Ouns) load pretrained
LIFT Honzon
Figure 10: Impact of the entropy coefficient and autoregressive loss horizon on finetuning per- formance (Booster T1, target speed = 1.5 m/s). Curves show evaluation episode return and body forward velocity. Larger a values degrade performance, leading to less stable convergence and lower final forward velocity. Using the pretraining gives reasonable results, while smaller yields more stable learning. Horizon sometimes fails to reach the target velocity, while horizons = 2 and ensure stable convergence, indicating that multi-step autoregressive prediction improves world- model training and stabilizes policy finetuning.
17

---

Published as a conference paper at ICLR 2026
A.5
PRETRAINING EXPERIMENTS
(a) TLLLoDDillyssiikFFlllTenii
(b) TiLowDDinloysiccoookhTTeemnn
(c) Fl
(d) TlloystickRouggherran
(d) GlloystickFllfTeraain
(f) G LoyystcckRoughhTraaaa
Figure 11: Gaits of LIFT across different tasks in the MuJoCo Playground.
18

---

Published as a conference paper at ICLR 2026
A.6 REAL WORLD EXPERIMENTS
Figure 12: Sim-to-rea real reinforcement learning with LIFT.
Figure 13: fim-to-real gait comparison in indoor and outdoor environments.
19

---

Published as a conference paper at ICLR 2026
A.7 FINETUNING: G1 EXPERIMENTS
Figure 14: Sim-to-sim transfer and fine-tuning results for the Unitree GI in Brax with target velocity of 1.5 m/s. Top: Policy before fine-tuning, exhibiting instability and shuffling gait. Bottom: Policy after fine-tuning, demonstrating a stable, human-lin like walking gait with reduced torso pitch.
(ailg
G1 mik (Detbin-Demiaaiien))
mik (Ctesif-Dsttittiioo
Eyaldd Retum
Euraad Licar Welelt
M/ Voa
24
Essinonentt Steps(-19")
Ensiroment SiSNE-110)
Esstromnntt Steps/-10")
Easirounent Stope(1107)
(Dum))
Figure 15: Finetuning results the Unitree GI in Brax. LIFT consistently improves policy behav- ior and reward performance. At target speed of m/s, body oscillations are significantly reduced, while at 1.5 m/s, the policy initially exhibits unstable motion in sim2sim but, after finetuning, cessfully stands and walks.
20

---

Published as a conference paper at ICLR 2026
A.8 NON-LOCOMOTION TASKS
Our framework is built around learning a world model of the robot's state dynamics, so any task whose reward depends only on the robot proprioceptive state can, in principle, be handled directly. This includes different locomotion gaits, whole-body motion tracking, and balance under distur- bances, since as long as the reward can be computed from the state, the policy can be finetuned entirely through world-model rollouts.
Whole body motion tracking Tasks. To better demonstraste this potential, we extended our pretraining pipeline from velocity-tracking locomotion to BeyondMimic-style whole-body track- ing (Liao et al., (025). We reimplemented the observation and reward structure of BeyondMimic in JAX within MuJoCo Playground and used the Unitree motion dataset (LAFANI) pretrain whole-body tracking policy for the Unitree G1 humanoid. Video demonstrations are available on our project website. Due to the engineering effort required to reproduce consistent observation/reward definitions and contact handling across MuJoCo and Brax, and to integrate these components into world-model finetuning pipeline, were able complete only the pretraining stage within the current timeframe. We therefore present these whole-body tracking results as preliminary evidence of LIFT's broader applicability. This limitation is practical rather than conceptual: once the corre- sponding Brax environment is finalized or real Unitree G1 hardware is available, the same stage-(iii) finetuning procedure can be directly applied.
(a) fightl subje:c((AAFANI
(b) fightl subjecc((AAFAN))
jumpsl_ junpss_suheer22L sabjen2
(d) nun (ahjecr2(LAPANN)
(d) walk1 subjct)((AFANN)
Figure 16: Gaits of LIFT across different whole body motion tracking tasks in the MuJoCo Play- ground.
Object-centric Tasks. For tasks involving external objects (e.g., kicking a ball), the current frame- work would need to extended to also model object dynamics, for example by augmenting the state with object pose and using simple physical priors such as conservation of momentum. This feasible but requires additional engineering and environment design, so we explicitly leave it as future work. Likewise, obstacle avoidance and more complex navigation are more naturally han- dled at a higher level: once a reliable low-level tracking controller is available (as in our setup), a high-level policy can be trained output velocity or pose targets. potentially with its own high-level world model as explored in hierarchical model-based RL (Hansen et al. (0224). Extending LIFT to hierarchical and object- tasks promising direction, but beyond the scope of this paper.
21

---

Published as a conference paper at ICLR 2026
B TRAINING DETAILS
Algorithm The Finetuning phase of LIFT
Require: Pretrained policy pretrained world model wm, world-model replay buffer D um batch size B; SAC replay buffer saci maximum episode length Tep; number of training iterations nuf7frraii world-model rollout horizon H Ham-
1: while v not converged do Su = env.resetly
2:
for i 1 to Tep do
3:
Or ~ (+((8())
4:
81-11, rt+1 etw.step(ss az)
5: Dwm t- Dm Ufst ai, r++11
6:
Dsac +- sac Ust at, 8141, Te+1)
7:
end for
8: Sample data from D and finetune the world model using equation
9:
for 6- 1 to nummrrain do // numarain is linearly increased from 10 to 1000
10:
Sample a batch (Batch size B) initial state so from the the D wm
11:
for i <- 1 to Hmm do // Hmm is linearly increased from 20
12:
a: ~ (+(ss)
1441, ww..stepls. St; ue) Il predict the next state and reward via world model.
14:
Daac &- Dac U{st at, 8t41, 1
end for
16:
Sample the transition from the Daac, and update the SAC actor and critic.
end for
18:
end while
return
B.1 PRETRAINING DETAILS OF SAC
Training Objective We pretrain the SAC policy in MuJoCo Playground. The critics minimize the entropy-regularized Bellman residual:
(ee.,a..ee1: ( aet1 (01) - min Qú, (5 log No To (at+1 (( 1]]
1=12
(6)
where Qú, denotes the target crities. Transitions sampled from the replay buffer D. The actor is updated by maximizing the entropy-augmented Q-value:
Esr D. (s> log (a(a))))) min Qv. as),,
1=1,2
The entropy coefficient a is adjusted to match target entropy H via
E,,-D, ( (- (-oog To (a(a))))) Ht)..
(8)
Finally, the target critics are updated by Polyak averaging: t reb + (1 T) where 7 E (0,1) is the update rate.
Hyperparameter-Tunie
We employ the Optuna framework (Akiba et al. 2019) for systematic
hyperparameter tuning, leveraging the Samps for parameter exploration in combination with a terminate trials with poor early performance. The tuned hyperparameters include the number of gradient updates per step (UTD), replay buffer size, entropy coefficient a, discount factor 7, learning rates of the actor, critic, and temperature parameter a, batch size, reward scaling, number of parallel environments, and Polyak coefficient The optimization objective is the episode reward obtained during policy evaluation. On the TiLowD1 i inJoyst hours of hyperparameter search single NVIDIA RTX 4090 GPU. The results indicate that tuning reduces the convergence time from about seven hours to only half an hour. On the G1 GIJoyst ioystici ick and Joysticck tasks, carried out similar ten-hour search using eight NVIDIA RTX 4090 GPUs, which led to improved reward performance and more robust convergence. Table [ lists the tuned hyperparameters.
Pruner (built on top of a Successi veiv www.alligla........ wwwalliinnrrrrrr to automatically
tasks, we conducted approximately ten
22

---

Published as a conference paper at ICLR 2026
Table 1: LIFT Hyperparameter Configurations for MuJoCo Playground Environments
Hyperparameter
TiLow
G1/T1 Joystick*
TlJoystickFlatTTrrraa
Parallel environments
1,000
4.096
4.096
Batch size
1,024
16,384
16,384
Gradient updates/step
9
19
16
Discount factor (7)
0.987
0,982
0.982
Reward scaling
1.0
16.0
32.0
Soft update (7)
0.024
0.002
0.007
Actor learning rate
1.03e-4
1.70c-4
2.05e-4
Critic learning rate
1.00e-4
1.88e-4
1.37e-4
Alpha learning rate
9.97e-3
6.44e-4
6.41e-4
Target entropy coefficient
0.50
0.40
0.17
Initial log
-3.35
-3,00
-6.05
Policy hidden layers Q-network hidden layers Activation
(512, 256, 128) (1024, 512, 256) swish
(512, 256, 128) (1024, 512, 256) swish
(512, 256, 128) (1024, 512, 256) swish
Max replay size (x10%) Min replay size Observation normalization
1.0 8,192
1.0
1.0
8,192
8,192 True
True
True 1
Action repeat
1
B.2 TRAINING DETAILS OF WORLD MODEL
(a) From Privileged State Brax Generalized State. We build the generalized coordinates/velocinn of Brax using the privileged state and denote them simply by (gt, 9:) for the rest of the section:
q: PP,, Thy gn.e), q. u wp, 9,,4) u." R(n)) o.
Pt (0, hz);
(9)
(10)
where R(n) S3) be the rotation matrix from the base quaternion (body-wooldd,, with R(g)T its inverse. Brax exposes differentiable rigid-body dynamics primitives- matrix M (g), Corio- lis/centrifugal terms C(g, (). and gravity G(q)-thereby avoiding any reimplementation of dynam-
ics and keeping the entire rollout-loss pipeline end-to-end diffcrentiable.
(b) From Privileged State and Action to Torque (repeat N substeps). The policy action is converted to motor torques T²" via a PD controller: 7tth Kp (a+ deffuult -qj,z)+ + K₁ (0- where K, and Ka are stiffness and damping: qf default default/standing joint position. The external torque TE come from the network (sp az). We assume the control rate feul is lower than the simulator/collection rate faim. Hence each control interval contains N faim simulator substeps of size Ataub, so that the control interval length At N Atsub.
Jest
(c) One Semi-implicit Euler Step (repeat N substeps). For substep k = O,...,N N with step size Abaub
q. M - (q:) 17 + of C(g.. G(g.)),
(11)
91 Maab 9: + Atsub Cc:
(12)
9j,t+ 9j.t + qi+Atanb
(13)
PitAtabb Pt + Atsub 2 Atums
(14)
B+ normalize n4 @ quat( wb w | Ataub)).
(15)
Here, @ = w/|| is the unit rotation axis; quat(@,0) 0) = [cos( ()) W sin( ()) is the axis-angle increment unit quaternion: @ denotes quaternion multiplication; normalize()) re-unitizes the quaternion to avoid numerical drift. After N substeps, we obtain:
a+ [Pt+At T4S b..4d)
(16)
4 |UifA whtse (..+a).
(17)
23

---

Published as a conference paper at ICLR 2026
(d) From (gi+ar, to the Next Privileged State.:
Joints Position/Velocity: (gj,++ (j++De)-
Base Linear Velocity: uf+ R(me+ae) U V2tar
(+
Base Angular Velocity: W/+
(18)
Quaternion: Re+Ar
Body Height: h441r = Pi+Ai,
B.3 TRAINING DETAILS OF WORLD MODEL IN FINETUNING STAGE
Concretely, we sample length-/[+1 H++ sequences DHr - { (sp, ar, s411 ac+l st
B
bm where B is the mini-batee batch size and H is the loss horizon (e.g., H-4). Let 8,1+1 be the model's 's one- step prediction (obtained by differentiable physics given (s (b,z)), 6,,41 the target, and log of the model's elementwise log-van variancee We minimize regressgee diagonal Gaussian negative log-likelihood over an H H-step unroll:
-tassial
B H-1
Cs = BH
1
(8,1-1 -8,411) 12,+11 o expp( - log )2,)) + log oi.
(19)
where @ denotes elementwise multiplication, E is the model's next-step prediction given (sh, (t,t), $h,+1 is the target, and log of is the elementwise log-variance, Gradients backpropa- gate through the entire H H-step unroll.
B.4 TRAINING DETAILS OF POLICY IN FINETUNING STAGE
A key advantage of physics-inforee informed design is that & exposes kinematic and dynamic signals that enable exact, simulator-consisse consistent reward computation without training a reward model. We thus define a per-step reward:
T++1 Σ wk re(Sf Ot, 8841)
(20)
with each re computed analytically. For example, foot orientation stability follows directly from each foot's roll angle and is penalized by
rell
Σ
(wd)
(21)
fE Right foot)
where of the roll of foot f extracted from the link quaternion in S. Table | lists the LIFT hyperparameters in finetuning.
24

---

Published as a conference paper at ICLR 2026
Table 2: Finetune Hyperparameter of LIFT
Hyperparameter
Actor-Critic
World Model
Policy hidden layers
[512. 256, 128] [1024, 512, 256]
Q-network hidden layers
World Model hidden size Activation
[400,400,40,4400
swish
swish
Learning rate
2e-4
le-3
Discount factor (?)
0.99
Batch size
256
200
Soft update (7)
0.001
Initial log
-7.13
Gradient updates/step
20
Real data ratio Init exploration steps
0.06
1,000
Replay buffer size
400k
60k
Reward scaling
1.0
Model trains/cpoch
1 1,000
Env steps/training
Horizon of exploration in world model
1 1-20 (epochs 0-10)
Gradient step per update
10~+1000 (epochs 0-+4)
Convergence criteria
0.01 (6 epochs)
Loss horizon
C ENVIRONMENT SETUP
C.I PRELIMINARY STUDY BOOSTER ENVIRONMENT
Task summary. The agent controls a 12-DoF T1 humanoid robot with simplified lower-body kine- matics. At each step, the agent outputs 12D continuous action that perturbs motor target positions relative to default pose.
State (39D). The state consists of torso quaternion (4), joint angles (12), base linear velocity in body frame (3), base angular velocity in body frame (3), joint velocities (12), gait phase represented by cosine/sine (2), gait progression (I), gait frequency (I), and base height (1).
Privileged state (39D). The privileged state uses the same 39-dimensional observation space as the state.
Action Space. The action space is continuous 12-dimensional vector, where policy outputs are constrained to the range [-1, 1]. Motor targets are computed using PD control:
K2 + qdefauh 9) Kalğses g)
(22)
where = 0.25 is the action scale factor. The proportional and derivative gains are defined as:
K₂ [200, 200, 200, 200, 50, 50, 200, 200,200, 200, 200, 50, 50] Ką 5,,
The gains are symmetric for both legs, with higher gains for hip joints (200 for K₁, 5 for K2) and lower gains for knee joints (50 for Ky. for Ka).
Reward Function Design. Uses multi-objection weighted reward function:
r: = Σ s.r(ss,, ac)
At
Control Architecture. Uses two-level control strategy:
1. High-level Policy: Outputs joint position increments, control frequency 100 Hz
25

---

Published as a conference paper at ICLR 2026
Table 3: Reward Function Components and Weights
Reward Term
Description
Default Weight
base height tracking.lin. lin_eel ref
Base height reward
0.2
Linear velocity tracking reward Reference trajectory tracking reward
1.2
3.0
Gait Generation. Phase-based gait controller:
Φ mod (6- fg: 1.0)
Left Leg Swing Phase 10 0.25| 0.5 T swingg
Right Leg Swing Phase = 1g - 0.75| < 0.5 - Tewing
where Tswing = 0.2s the swing phase duration.
Termination Conditions. Episode terminates under the following conditions:
Base height h & [0.3, m
Base linear velocity > 10.0 m/s or angular velocity > rad/s
Torso roll angle |6] > 7/4 4 or pitch angle [0] > x/4
Joint position velocity exceeds mechanical limits
Experimental Setup. In the Preliminary Study (Section A.1
Target forward velocity: 0.2 m/s
Main reward terms: swing leg reference, base height tracking, linear velocity tracking
26

---

Published as a conference paper at ICLR 2026
C.2 PRETRAINING ENVIRONMENTS
C.2.1 TILOWDIMJOYSTICK
(TLLowDimJeystick an example illustrate the pretraining environment setup, TlJoystick and GlJoysticc are omitted for brevity.)
Task summary. The agent controls 12-DoF T1 humanoid robot with simplified lower-body kine- matics. At each step, the agent outputs 12D continuous action that perturbs motor target positions relative to default pose. The environment returns noisy states and scalar reward. Episodes termi- nate upon falls or numerical errors (NaNs).
State (47D). The state consists of gravity vector in the body frame (3), gyroscope readings (3),
joystick command (Ugs f'y, w) (3), gait phase represented by cosine/sine (2), joint angles relative to the default pose (12). joint velocities scaled by 0.1 (12), and previous action (12).
Privileged state (110D). The privileged state includes all elements of the 47D state, together with additional raw sensor signals (gyroscope, accelerometer, gravity vector, linear velocity, global an- gular velocity), joint differences, root height, actuator forces, contact booleans, foot velocities, and foot air time.
Action space. The action is continuous 12-D vector; policy outputs lie in [-1, Motor targets are computed using a PD control law:
u = Ky(a + Idefault - q) + (a 4)
(23)
where u is the output motor torque, Kp and Ka are the proportional and derivative gains, respec- tively, a is the action output from the policy Qdefaul nominal joint position, q is the current joint position, and é is the current joint velocity.
Reward Instantaneous reward: rt str at), where 8; are term weights and At 0.02s.
Table 4: Default reward terms (Note: Ula, b] denotes uniform distribution over [a, b))
Term
Description
Scale
survival
Survival bonus per step. Track commanded v,, velocity. Track commanded eg velocity. Track commanded yaw rate. Reward proper foot swing phase. Penalize deviation from target height (0.68m).
0.25
tracking_lin.co _ei_vel_x tracking liiinglli_vvee lin_velyy tracking -ank_veg feet swing base height orientation
1.0
1.0
2.0
3.0
-20.0
Penalize torso tilt from vertical.
-10.0
torques
Penalize large torques.
-1.0e-4
torque_tiredness
Penalize torque near limits.
-5.0e3
power
Penalize positive mechanical power.
-1.e-3
lin_vel.z
Penalize vertical base velocity.
-2.0
ang vel xy
Penalize torso roll/pitch rates.
-0.2
dof vel
Penalize joint velocity.
-1.0e-4
dof_acc
Penalize joint acceleration.
-1.0e-7
root_acc
Penalize root link acceleration.
-1.0e4
action_rate
Penalize rapid action changes.
-0.5
dof.posslimite limits
Penalize joint limit violations.
-1.0
collision
Penalize self-collissons. Penalize slipping contacts.
10.0
feet feet_slip
-0.1
feet_roll
Penalize foot roll angles.
-1.0
feet_yaw_dif
Penalize difference foot yaw angles. Penalize deviation from base yaw.
-1.0
fcet_yyw_mean
-1.0
feet.di distance
Penalize feet being too close.
-10.0
Implementation details
27

---

Published as a conference paper at ICLR 2026
Control period: ctrl.dt 0.02 simulation step: sim.dt 0.002 s.
External pushes enabled with interval 5.0-10.0s and magnitude 0.1-1.ON.
Enhanced tracking rewards with separate x/y linear velocity tracking.
Sophisticated foot kinematics penalties including roll, yaw, and swing phase rewards.
Table 5: Domain randomization parameters (Note: [a, b] denotes uniform distribution over [a, b],
(C)c) denotes uniform distribution over [-c,e])
Parameter
Range
Description
Floor friction
U/0.2, 0.6)
Ground friction coefficient
Joint friction loss
xU700.9 1.1]
Joint friction scaling
Joint armature
xU[, 1.05]
Joint armature scaling
Link masses
xU[0.98, 1.02]
Body mass scaling
Torso mass
+U[-1.0, kg
Additional torso mass
Initial joint positions
+0(0. rad
Default pose randomization
Joint stiffness
xU
Actuator gain scaling
Joint damping
0.
Damping coefficient scaling
Ankle damping
Higher range for ankle joints
C.3 FINETUNING ENVIRONMENTS
C.3.1 TILOWDIMJOYSTICC (FINETUNE)
(CILowDinJeystick an example illustrate the finetuning environment GlLowDimJoystic is omitted for brevity.)
setup,
Task summary. This is a simplified version of the TimowDDmJoystiic environment designed specifically for sim-to-sim transfer from MuJoCo Playground to Brax and finetuning in Brax. The environment ensures that policies trained MuloCo Playground can be successfully transferred to Brax with similar reward scales and performance characteristics. The reward settings are adopted from Booster Gym (Wang et al. 20255.
Key modifications for finetuning:
Normalized observations using predefined limits for consistent scaling across simulators
Simplified reward structure with many terms zeroed out to focus on essential behaviors
Continuous gait phase representation using gait process and frequency
State (47D, normalized). The normalized state consists of gravity vector (3), gyroscope readings
(3), joystick command (Vz, vg. w) (3), gait phase cosine/sine (2), joint angles (12), joint velocities (12), and previous action (12). All observations are normalized to [-,, using predefined limits.
Privileged state (87D, normalized). Includes base state plus raw gyroscope, gravity vector, ori- entation quaternion (4), base velocity, joint positionss/ellocites,, root height, gait process, and gait frequency.
Observation normalization: Observations are normalized using:
2 (obs min)
obs normalizee
max min
with predefined limits for each observation dimension.
Action space. Motor targets are computed using PD control law:
(24) where is the output motor torque, Kp and Ka the proportional and derivative gains, respec- tively, a is the action output from the policy Qdefauh is nominal joint position, q is the current joint position, and is the current joint velocity.
u = Kapa (a + Adefault - q) + (a -
28

---

Published as a conference paper at ICLR 2026
Table Finetune reward terms (Note: many terms disabled for sim-to-sim transfer)
Term
Description
Scale
survival
Survival bonus per step.
0.25
tracking_lin lar_g_li__lllllaaeee _in_vel_x tracking Laipgg_lii_eeee ______nvlllll vel_y tracking_ane -ngg-veagee
Track commanded ez velocity. Track commanded vy velocity. Track commanded yaw rate.
1.0
1.0
2.0
base height
Reward proximity to target height (0.65m).
0.2
orientation feet swing feet slip feet_yaw_die syff
Penalize torso tilt from vertical.
-500
Reward proper foot swing phase.
3.0
Penalize slipping contacts.
-0.1
Penalize difference in foot yaw angles.
-1.0
feet-yaw .eean
Penalize deviation from base yaw.
-1.0
feet_roll
Penalize foot roll angles.
-1.0
feet distancee
Penalize feet being too close.
-10.0
Reward terms (simplified for transfer):
Disabled reward terms: torques, torque torque tiredness, power, lin.vel.z, ang-vel.xy. dof.vel, dof,ace, root_ac,, action_rate, dof lof.pos_limits collision, feet_vel_z (set to 0.0 scale).
Implementation details
External pushes enabled (interval: 5.0-10.0s, magnitude: 0.1-1.ON)
Normalized observations ensure consistent scaling across simulators
Table 7: Domain randomization parameters (Note: Ula, b] denotes uniform distribution (a, b).
(C) denotes uniform distribution over [-c,e]
Parameter
Range
Description
Floor friction
U 0.6]
Ground friction coefficient
Joint friction loss
(.,, 1.1]
Joint friction sealing
Joint armature
x., 1.05]
Joint armature scaling
Link masses
xU 1.02]
Body mass scaling
Torso mass
1.00 kg
Additional torso mass
Initial joint positions
=U[0.05] rad
Default pose randomization
Joint stiffness
xU/0.7,1.3
Actuator gain scaling
Joint damping
xU
Damping coefficient scaling
Ankle damping
xU
Higher range for ankle joints
D LARGE LANGUAGE MODEL. USAGE
We gratefully acknowledge the of ChatGPT and DeepSeek assist in polishing, refining, and condensing the text of this paper.
29


Reviews:
[{"review_id": "xoe5ysWiVm", "reviewer": "Reviewer_2CYj", "summary": "This paper proposes LIFT, a three-stage framework to bridge the gap between large scale pre-training and sample-efficient fine-tuning for humanoid locomotion control.\n\nThe framework first challenges the consensus that PPO is the default for large-scale training. It demonstrates that off-policy SAC, when implemented in JAX with large-batch updates and a high Update-To-Data (UTD) ratio, can achieve comparable or faster wall-clock convergence than PPO for pre-training, while also enabling zero-shot sim-to-real.\n\nThe authors then propose a model-based fine-tuning stage, where a physics-informed world model (combining Lagrangian dynamics with a learned residual network) is pre-trained on the SAC simulation data. In stage (iii), this world model is adapted to a new environment. The robot collects rollout data in the target environment using a deterministic policy for safety. This new data is used to fine-tune the world model. Stochastic exploration is then confined to rollouts within this updated world model, and the policy is updated using this safe, synthetic data.\n\nThe authors validate this approach in \"sim-to-sim\" experiments, showing that the LIFT framework can successfully fine-tune policies for new, out-of-distribution tasks (e.g., new velocity commands in a new simulator) where model-free fine-tuning of SAC, PPO, and FastTD3 all fail.", "questions": "The major questions are addressed in the weakness section. Here are some additional questions:\n\n*  What are the primary practical barriers that prevented the authors from attempting the stage (iii) *fine-tuning* loop on the real robot? How would you propose to manage the safety of the initial \"deterministic\" data collection in the real world?\n* In stage (iii), the world model and the policy are fine-tuned concurrently (Sec 4.3). How are these updates interleaved? Is there a risk of the policy overfitting to an inaccurate, partially-fine-tuned world model, and how is this instability managed (e.g., how is the world model rollout horizon scheduled)?\n* In Section 4.1, the paper states it relies on SAC's state-dependent variance for exploration. How critical was this choice during pre-training compared to a simpler state-independent variance? Did this choice have any downstream effects on the fine-tuning stage (e.g., better exploration *within* the world model)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper provides a valuable finding by demonstrating that SAC, an off-policy algorithm, can be successfully scaled for massively parallel pre-training of humanoid controllers.  The open-sourcing of a JAX-based SAC implementation for this purpose is a welcome contribution.\n* The proposed hybrid physics-informed world model works and the target environment adaptation works well for humanoid simulation. The ablation study in Appendix A.3 (Fig. 6) shows that a standard \"black-box\" MBPO-style model completely fails to adapt, while LIFT's physics-informed model succeeds. This provides strong evidence that embedding physics priors is essential for world-model-based adaptation in this domain.\n* The LIFT framework's main contribution is its integration of pre-training with a model-based fine-tuning loop. The proposed world-model finetuning framework could be used in more humanoid finetuning cases.", "weaknesses": "* Lack of Real-World Fine-Tuning (Sim-to-Sim): The paper's primary weakness is that its core claim—safe, efficient fine-tuning—is only validated in a \"sim-to-sim\" setting (MuJoCo to Brax). While the paper shows zero-shot *pre-training* on a real robot (Appendix A.6), it does not \"close the loop\" and test the *fine-tuning* procedure in the real world. The claim of \"safety\" is significantly weaker without this, as collecting even \"deterministic\" data in a new real-world environment carries risks (e.g., from initial state distribution mismatch or model error) that the paper does not address.\n\n* Missing Discussion on Safety for Real-World Adaptation: Following W1, the paper overlooks the practical hardware challenges of data collection for fine-tuning. Even a deterministic policy can fail. Methods like RTR [1] have shown that specially designed safety systems (e.g., a \"teacher\" robot arm) are crucial for enabling this kind of real-world adaptation on humanoids. A discussion of these practical requirements is missing.\n\n* Incomplete Related Work and Missing Citations: The related work section is sparse in several key areas, weakening the paper's positioning and claims of novelty. (a) Context for SAC Pre-training: The paper's first contribution is scaling SAC for pre-training. However, it fails to cite work that specifically discusses the *challenges* of scaling SAC in massively parallel simulators and its typical instability compared to PPO like [2]. (b) Model-Based Fine-Tuning Landscape: The paper's core contribution is a model-based fine-tuning method, but it omits citing some other model-based adaptation strategies, such as learning residual \"delta action\" models (e.g., ASAP [3]) or other concurrent humanoid world-model papers (e.g., Hu et al., 2025 [4]). I believe a whole paragraph of this line of related work is needed.\n\nReferences:\n\n[1] Hu, Kaizhe, et al. \"Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids.\" Conference on Robot Learning (CoRL). 2025. \n\n[2] Raffin, Antonin. \"Getting SAC to Work on a Massive Parallel Simulator...\" Blog Post. 2025. \n\n[3] He, Tairan, et al. \"ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills.\" Robotics: Science and Systems (RSS). 2025. \n\n[4] Xinyang Gu, et al. \"Learning Humanoid Locomotion with World Model Reconstruction.\" Robotics: Science and Systems (RSS). 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RZdHTCrNkz", "reviewer": "Reviewer_JXM1", "summary": "This paper introduces LIFT, a three-stage framework that bridges large-scale off-policy pretraining and efficient safe finetuning for humanoid robot locomotion. The key pipeline involves: (i) large-scale pretraining of policies using Soft Actor-Critic (SAC) in massively parallel simulators, (ii) pretraining a physics-informed world model leveraging Lagrangian dynamics and residual neural predictions on stored transitions, and (iii) iteratively finetuning both the policy and world model in new environments using deterministic real-world execution and stochastic exploration confined to the model. Experiments on multiple humanoids and transfer scenarios indicate that LIFT achieves competitive pretraining performance and robust, efficient finetuning—outperforming prior model-free and model-based RL baselines under challenging adaptation conditions. The code is open-sourced.", "questions": "1. Can the authors clarify their design choices regarding the length and scaling of the autoregressive unroll horizon in world model training? How sensitive is LIFT to this hyperparameter in practice?\n2. In real-robot adaptation, what additional safety or recovery mechanisms (beyond physics-consistent reset) would be paramount, and how does LIFT plan to address distributional shift that may be more severe than sim-to-sim transfer?\n3. Is there a plan or preliminary evidence for multi-task or non-locomotion humanoid adaptation under LIFT, or does the framework generalize only to velocity-tracking locomotion?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed pipeline is systematically motivated, decomposing the challenge of bridging data-hungry RL pretraining with safety and efficiency in real-world adaptation. \n- The integration of Lagrangian dynamics with learned residuals for improved world model rollouts is novel in the humanoid domain and addresses both sample efficiency and stability concerns. \n- The work provides results for two complex humanoid platforms (Booster T1 and Unitree G1), encompassing flat and rough terrain, various velocity targets (in-distribution, long-tail, and out-of-distribution), and sim-to-sim as well as sim-to-real transfers. Notably, the policy achieves zero-shot deployment and robust adaptation.", "weaknesses": "- **Task Diversity**: The evaluation focuses on forward locomotion tasks (varying speed targets on flat or rough terrain). This is a narrow slice of humanoid skills. The significance would be further bolstered by testing, say, different locomotion gaits or disturbances, or tasks like turning, obstacle avoidance, etc. It’s unclear how readily the approach extends to non-locomotion tasks or more complex objectives.\n- **Use of Privileged Information**: The method relies on a privileged state (including full physical state and other quantities like ground friction) for training the world model and critic. This is standard in sim-to-real pipelines, but it presumes access to those privileged observations. The paper doesn’t fully discuss how feasible this is outside simulation – e.g., obtaining exact friction or global state on a real robot might require estimation. If such info is unavailable or inaccurate, it could affect performance. The authors do not address this point in depth, and it could be a concern for deploying the finetuning part in the wild.\n- **Comparison to Other Model-Based RL**: The paper compares to MBPO as a baseline for learned dynamics, but other model-based approaches (e.g., Dreamer or recent model-based fine-tuning methods) are not explicitly compared in finetuning. Given that DreamerV3 and others have shown strong model-based learning on locomotion, an empirical comparison or at least a discussion would have been useful to position LIFT against the broader landscape of model-based RL in terms of performance and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Wk7yUQE7o", "reviewer": "Reviewer_2rhQ", "summary": "The authors address issues with reinforcement learning control of humanoid robotic systems. They point out that proximal policy optimization can be brittle when transferred to zero-shot settings and is sample-inefficient even in parallel environments. The proposed solution is to use the soft actor-critic algorithm with a learned physics-aware world model to first pre-train a policy on a large-scale, parallelizable simulator and then finetune online using synthetic samples generated by the world model for the target humanoid task. The authors demonstrate that their pretrained policy can be deployed zero-shot in the real world and performs better during finetuning on simulation tasks.", "questions": "Q0: Could the author's expand on the following interpretation (Line 416): . PPO maintains reasonable performance initially but degrades over time, eventually collapsing, likely due to its KL-regularized updates that limit fast policy adaptation? Did the author's include the KL divergence between the current policy and old policy? What about the clip coefficient? \n\nQ1: For Q1, if your algorithm only does just as well as baselines, what inherent advantage does LIFT provide that prior methods do not have for the pretraining phase? I'm thinking about algorithmic benefits, not the transfer learning aspect (i.e. Q2).\n\nQ2: Line 211 - Are the authors actually using a Gaussian policy, or is there a tanh activation transformation to normalize actions? This transformed distribution was used in the original SAC. \n\nQ3: How much of the world model is learned as opposed to being defined by the BRAX? What simulator is used to pretrain the model? Is it re-using the same tools as the world model simulator? Is there any issue of mismatch between the pre-trained simulator and the target environment? \n\nQ4: Why is SSRL not included as a comparison in the reported results of the paper for Q1 or Q2? The author's algorithm appears heavily motivated by this prior work, yet no results appear in the main paper.\n\nQ5: What are the reasons for not comparing LIFT with a physics-informed model vs a non-physics-informed model?\n\nQ6: What are the potential limitations of LIFT for high-dimensional observation data? What changes would be needed to use images, for example?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "> The authors provide a fast, scalable implementation of SAC, which opens up new avenues for research on the limitations of online reinforcement learning\n> The authors conduct extensive studies to verify the efficacy of their algorithm. I appreciated their use of 8 seeds, which is much better than the typical practice in other deep RL research (as few as three seeds in most works I have reviewed). The inclusion of zero-shot transfer in real robotics was also a promising finding, although no experiments were conducted to finetune on real-robot data.\n> There are interesting findings to this work, such as identifying the limitations of prior research on plasticity, finding that the pretraining algorithm does affect finetuning performance.", "weaknesses": "Our biggest concern is the quality of organization and writing in the current submission. A good example of this is the exclusion of results to answer Q3 in the experiment section. If the paper had reduced content in other sections, it would have been easier to include them in the main paper. For example, Comments on design choices in the method section can be moved to the appendix, such as those about the experimental evidence used to justify decisions, which should be explained in the Q3 results (lines 256 & 257). The introduction could be more coherent and more transparent about why physics-aware world models are being employed.\n\nIn terms of contribution, this type of work might have more impact at a robotics-focused conference. The authors' method seems promising as a pretrain-and-finetune solution for humanoids, but it offers limited novelty beyond addressing issues with SSRL, which their work is heavily influenced by. There are aspects of the work that challenge this comment, such as their finding that issues of plasticity (lines 223-227) do not arise when scaling the parallel environments, but this is not emphasized in the main experiments and is left as future work. \n\nThe experiment section demonstrates that the framework works and identifies which hyperparameters the model is sensitive to. I did not find the results shown for Q1 particularly insightful, because all that is shown is that the authors do \"just as well\" as other methods. If there are other benefits, this should be emphasized; otherwise, Q1 could be moved to the appendix or condensed to make space for more critical analysis of LIFT. Q2 & Q3 deserve more discussion or consideration, particularly as Q2 is the application scenario the authors are studying, and Q3 helps the reader understand which hyperparameters are essential for applying LIFT. \n\n\n\nWriting opinions \n\nLine 063: The last sentence in the opening paragraph doesn't logically follow from the limitations of PPO per se, particularly because the \"pretrain-finetune paradigm\" isn't explained as to WHY it solves PPO issues. Maybe something along the lines of \"off-policy algorithms are a means of addressing this\" would make more sense given the following paragraph.\nLine 065: The author uses \"However\" twice in this paragraph.\n\nLine 076: \"we conduct a preliminary study in which humanoid walking is trained from scratch while limiting data collection to deterministic execution.\"\n> This sentence should be deleted. When I read this in the introduction, my first question was whether the research was complete. That's probably not what the authors meant, but that is how it came across.\n\nLine 082: (ii) The use of physics-informed world models is not motivated well in the introduction, so from the reader's perspective, this comes out of nowhere as to why these are used over non-physics-aware models. \n\nLine 238: The information on the state vector could be moved to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)