Accelerating Visual-Policy Learning through Parallel Differentiable Simulation
Authors: Haoxiang You
Yilang Liu
Ian Abraham
Venue: NeurIPS 2025 spotlight ()
Keywords: visual policy learning
analytical policy gradient
differentiable simulation
Primary area: reinforcement_learning
TL;DR: We introduce a computationally efficient method for visual policy learning using differentiable simulation.

Abstract: In this work, we propose a computationally efficient algorithm for visual policy learning that leverages differentiable simulation and first-order analytical policy gradients.
Our approach decouple the rendering process from the computation graph, enabling seamless integration with existing differentiable simulation ecosystems without the need for specialized differentiable rendering software.
This decoupling not only reduces computational and memory overhead but also effectively attenuates the policy gradient norm, leading to more stable and smoother optimization.  
We evaluate our method on standard visual control benchmarks using modern GPU-accelerated simulation. 
Experiments show that our approach significantly reduces wall-clock training time and consistently outperforms all baseline methods in terms of final returns.
Notably, on complex tasks such as humanoid locomotion, our method achieves a $4\times$ improvement in final return, and successfully learns a humanoid running policy within 4 hours on a single GPU.
Videos and code are available on https://haoxiangyou.github.io/Dva_website

Accelerating Visual-Policy Learning through Parallel Differentiable Simulation
Haoxiang You
Yilang Liu
Department of Mechanical Engineering Yale University
Department of Mechanical Engineering Yale University New Haven, CT 06520 yilang. www.g..lu@yall..om edu
New Haven, CT 06520 haoxiang. youlyaloouylleeeeeo
Ian Abraham
Department of Mechanical Engineering Department of Computer Science
Yale University
New Haven, CT 06520
ian. abraham@yale edu
Abstract
In this work, propose computationally efficient algorithm for visual policy learning that leverages differentiable simulation and first-order analytical policy gra- dients. Our approach decouple the rendering process from the computation graph, enabling seamless integration with existing differentiable simulation ecosystems without the need for specialized differentiable rendering software. This decoupling not only reduces computational and memory overhead but also effectively attenu- the policy gradient norm, leading more stable and smoother optimization. We evaluate our method on standard visual control benchmarks using modern GPU-accelerated simulation. Experiments show that our approach significantly reduces wall-clock training time and consistently outperforms all baseline methods in terms of final returns. Notably, on complex tasks such as humanoid locomotion, our method achieves 4x improvement in final return, and successfully learns a humanoid running policy within hours on single GPU. Videos and code are available on htttp:://haxiiangyol gitbub. io/Dva_wessite/
Introduction
Learning control robots from visual inputs is key challenge in robotics, with the potential to enable a wide range of real-world applications, ranging from autonomous driving and home service robots industrial automation. Most methods for learning visual policies fall into two categories; imitation learning and reinforcement learning (RL). Imitation learning trains policies by mimicking expert demonstrations, which are typically collected via human operation [Bojarski]
et al. 2016 Kendall et al., 2019] or teleoperation systems [Chi et al.] [024 Black et al., 2024]. When expert demonstrations are scarce or difficult obtain, visual policies can instead be learned through RL [Mnih et al. [0115] Hafner et al.) [019]. However, RL methods typically require long training times and substantial computational resources, such as large-scale GPU clusters, to achieve effective
control.
Recent advances in differentiable simulation have enabled alternative policy optimization methods, known as analytical policy gradients (APG) [Freeman et al., 202] Xu et al. 2021 Schwarke et al. 2024
These methods achieve much higher computational efficiency by replacing zeroth-order
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Fast training of humanoid running policy from pixel input. Our method learns stable running gait in 4 hours on a single RTX 4080 GPU.
Figure I:
gradient estimates with first-order gradients. However, extending APG methods to visual control remains challenging: obtaining high-quality differentiable rendering is non- non-trivial, and computing Jacobians over pixel inputs is both memory- and computationally inefficient. Consequently, existing
approaches either train separate differentiable renderers [Wiedemann et al. [2023] Liu et al.) [2224] or
rely on low-dimensional visual features, which demand considerable engineering effort [Heeg et al.,
2024 Luo et al. 2024..
In this work, we propose Decoupled Visual-Basee Analytical Policy Gradient (D. Va), novel method for learning visual policies using differentiable simulation. The core idea is to decouple visual observations from the computation graph, eliminating the need differentiate through the rendering process. We find that this decoupling not only improves memory and computational efficiency by avoiding Jacobian computations over pixel space, but also normalizes the policy gradient, making visual policy learning more stable. We further provide a formal analysis of our new computation graph, demonstrating that the proposed decoupling policy gradient can be interpreted as a form of policy distillation from open-loop trajectory optimization. This reveals a fundamental connection between open-loop trajectory optimization and closed-loop policy learning.
Finally, we benchmark a diverse set of visual policy learning methods using a GPU-accelerated simula- tion platform that supports parallelized physies and rendering, providing a robust and scalable testbed for evaluation. Our comparisons include the proposed D. Va, two model-free RL algorithms [Laskin al. 2020a Yarats et al. 202T]. the model-based RL method DreamerV3 [Hafner al., [2023] an analytical policy gradient method with differentiable rendering, and tovesal distillation MMu al. [2025] a two-stage framework that first trains a state-based policy and then distills into a visual policy through imitation learning. Experiments highlight D. Va's superior computational efficiency across wide range of control tasks.
In summary, our contributions are: (a) Proposing Va, a computationally efficient method for visual policy learning; (b) Benchmarking diverse visual policy learning approaches state-of oftte----he-rrr art simulation platforms: (c) Providing an analysis of analytical policy gradients and highlighting the new opportunities for integrating policy learning with trajectory optimization techniques.
2
Background
In this section, we formally define the policy optimization problem and introduce key concepts in analytical policy gradient methods.
2.1 Problem formulation
We consider the dynamical system 8 fSr az), where S, E S denotes the state and a, E A the action. The dynamics function / S A S is assumed be fully differentiable with respect to both state and action. Let o₂ = g(s²) E O denote the observation, where S + O is the sensor model. Throughout this paper, we assume s; a low-dimensional internal representation of the robot (c.g., joint states), while represents observation derived from Sy (c.g., images).
{So, ap, $1, a .. Srr......,,, ar) which is sequence of state-action pairs with horizon T. The total return is defined as [(r = Ei= 7" R(s;, ar), where 7 E (0, 1) is the discount factor, and R S x A + R is the reward function. We denotes the discounted temporal reward as r. = 7 R(s,, a.). A feedback policy =--oo,, 0) x θ - A(A) family of conditional probability distributions that maps observation to a probability distribution over
Consider the trajectory

---

actions. Typically, this distribution is modeled as Gaussian, allowing the action to be expressed via the reparameterization trick: a, = ju(o,, Q) + (o, 0) Ee where μ x A, O x A represent the mean and standard deviation respectively, and ~ (0,, I) is injected noise. Given an initial condition Sg, and sequence of injected noises = {Eo, E1....6r). a trajectory T can be generated by rolling out from policy under dynamics f and sensor model g. which we explicitly denoted as T(s. 0, E) and the corresponding return as J (T 0, ). For notational simplicity, we omit the explicit trajectory т and write the return directly as J (sa. 0. E) The expected return for given policy is defined as V(Ø) Eso V4 (((,)) [So,
T 0, E), where Pu is initial distribution.
The goal of policy optimization is to find policy parameters maximizing the expected return.
2.2 Analytical policy gradient
Here, we provide background on analytical policy gradient (APG) methods. These methods compute the policy gradient as
VoV= Po (O..)
VoJ (sn, 0, Esoorpo E ((
V ot
(I)
where the gradient of each term in the sum is given by
Dr. da:
Ôr, ds:
Vorz
+
t 0,...,
oa
d0
Üs; d0
da:
Da, do, ds,
+
t == O,...,T
d0
d0
0o, ds, d0
ds; Üs,
ds Us.
da--1 t= 1...7 T and 0.
dso
d0 Ús d0
Da
d0
de
Here, 5 denotes the total derivative, and @ represents the partial derivative, both expressed in matrix form as Jacobians. The expectation can then be estimated via empirical sum. As these methods estimate the first-order policy gradient by backpropagation through trajectories, they are also named as first-order policy gradients (FoPG) or backpropagation through time (BPTT).
Short-horizon actor critic (SHAC) The trajectory gradient, i.c., VeJ (so. O, E), can quickly become intractable as the horizon 7 increases due to exploding gradients by multiplying series of matrices. The exploding gradient of an individual trajectory leads to the high-variance variance empirical estimate of the policy gradient [Metz al. [02T], as well as an empirical bias problem (Suh et al. 2022]. Fortunately, these problems are largely solved by the Short- Horizon Actor-Criti (SHAC) method [Xu et al. 2021]. The key idea is to truncate the long trajectory into smaller segments and incorporate learned value function for long-horizon predictions. More specifically, at each iteration, the SHAC algorithm optimizes the following actor loss
N
1o+h-1
L. =
(1) alss R(s)
22)) +
(3)
Nh
t=fa
(
where 8/ and a are states and actions of the i-th trajectory rollout, and Va : S is the value function learned with TD-À tricks [Sutton et al. 1998
3 Method
In this work, we extend SHAC to handle complex observations, e.g., images, whereas the original
SHAC primarily focused on low-dimensional state spaces. The main challenge in visual setting is
calculating the observation Jacobian do, which is both computationally and memory intensive due
da: to the high dimensionality of images. To address this, we omit all terms of the form (Ba dos deg (Bo d0 )
resulting in quasi-policy gradient that we refer to the decoupled policy gradient (DPG). In this
section, we provide formal analysis showing how the DPG works, as well as algorithm that utilizes DPG for training visual policy.
3

---

Decoupled Policy Gradlent
Analytical Policy Gradient
OD
ST
Shared Policy Parameters
Reward
Forward
Backward
Backward Cutoff
Figure Computation graphs of decoupled policy gradient (DPG) and APG. The policy gradient is traced from any reward rt. propagated backward through the graph to the shared parameters APG backpropagates through the entire pipeline, whereas the DPG prevents gradient flow through the rendering process.
3.1 Decoupled policy gradient
We divide the analytical policy gradient ① into two parts by separating all terms involving dos from dec
those that do not: VoV= VoV + B, where
T Σ
VoV- = ExO es ((N(O
VoJ 0, = Rsorpo E
V
or
t=
dr Da,
Ôr: dsc
Ver:
O...T
0a 00
Üs, d0'
ds:
as.
ds
Ōs:
oa:-1
and dso
1= 11.....
0,
(4)
d0
Ús:-1
d0
da:-1 00
d0
and
T
dr. da, do, d&c
Exo
Σ
Oa, dor ds, )
4 N(O.))
1=
ds:
0s: δai-1 1
Ōa,-1
doe ds + 0s d0
Ss: dsi-1 1...
dsn
T and
0. (5)
d0
0ai-1 00
00,-1 ds d0
d0
Detailed derivation of this decomposition is provided in Appendix A We refer to VoV 4 as the decoupled policy gradient, which improves the policy by distilling results from open-loop trajectory optimizations. We denote (5) as control regularization which captures the interdependence between actions. Below, provide conceptual explanation of each term, describe the rationale behind their naming, and validate the effectiveness of the decoupled policy gradient through experiments.
Distilling from open-loop Trajectories Here, show how the decoupled policy gradient can be interpreted as distilling from open-loop trajectories. We begin by initializing a open-loop sequence of controls, A {ag, . ar), by rolling out the policy (-|--; 0) under the initial condition Sp. Given the open-loop control sequence A and initial condition So. the sequence of states {so, ..... , ST and observation O {Op, Oi,.... OT } can be reconstructed via dynamics and sensor model g. In this case, the return (sp, A) Σ Tt solely a function of the initial condition and the control sequence A. The gradient of return with respect to the control sequence is given by
T
VAJ = [Van J, Jn J..., T), where Vn =
Σ j=t
V
Tj and
a,
Or, ;s; 08,-
2 06441 da
when >
(6) We improve the control sequence by taking small step B in the gradient direction for each action
de, De; 0s; dr: Ga.
1= .....
when j =
a, a: + J.
We denote the updated sequence as aa. ai,.... )r). The behavior cloning loss is then defined the discrepancy between the actions generated by the current policy and those in the updated

---

control sequence:
T
C,c = 23 t=0
|n(o 0) + ( 0) 0 a 12
(8)
where E = {Eo ET)
) is same injected noises use to initialize open-loop sequence A.
Theorem 1. The decoupled trajectory gradient in 4 equals the negative gradient of the behavior cloning loss in Equation ( i.e., VoJ (su, 0, - 0, A,
Proof. Given that the observation sequence O generated by rolling out the policy, we have a, [1(0;; 0) (0,; 0) o 6x. Therefore, the gradient of the behavior cloning loss simplifies to
8a.
Σ 1=
(ar āz)
(9)
00
Substituting E.q. 7 into resulting in
T
T
dat
VeLsc(0,0,A,, Σ 1=0
(-
vV
Σ 1=0
V J)
(10)
00
00
Finally, substituting V from E.q. into (9) and rearranging terms completes the proof.
Theorem T highlights the close connection between feedback policy optimization and open-loop
trajectory optimization. Iteratively applying gradient ascent with decoupled policy gradient (4) can be interpreted as alternating between two stages: (1) generating trajectories by rolling out the current policy and improving them through trajectory optimization, and (2) distilling the optimized
trajectories back into the policy.
Cosine Similarity
Gradient Norms
100
APG
DPG
Control regularization We discuss the dropped bias (5), which involves adding multi- ple terms of the form (8r, da, do, (da; do, ds, do ). Here,
0.0
Ors captures how the reward timestep t Da,
-1.0
2020
4050
2000
1003
Returns
Returns
changes with the action a,, while the term (aa do, ds, Do, ds, dØ reflects how past trajectory in- fluence current decision making, i.e., a, Al- together, the bias term (5) quantifies how the previous experiences influence current actions via coupling through the shared policy. and thereby impacts long-term return. In contrast, the decoupled policy gradient ( captures how the current action affects future states, but ig- nores the interdependence between actions.
Figure 3: Comparison of APG V and DPG V on Hopper with full state observation. Both gradients are computed from the same of 0 values collected during SHAC training. The x-axis shows the return for cach Left: Cosine similarity is generally positive, indicating V V is valid ascent direction. Right: Gradient between APG and DPG. In this experiment, conducted state space where g = identity, the control regulariza- tion term B acts as residual connection. As result, APG generally exhibits smaller
compared DPG.
Figure compares the APG V and DPG Vo both computed with respect to the thort-horizon actor loss 3 The cosine similarity between the two gradients is positive in most cases, indicating that VoV generally provides valid ascent direction for policy improvement. Another noteworthy observation is the difference in norm between the full analytical policy gradient VoV and our sasii-pooiyy-rradieee estimate, DPG V.V. When we conduct the experiment on state space, i.e.. o, iddenitt(ss)) 8t, the additive control regularization (5) operates acts similarly to residual connection within the computation graph. The additive residual connection contribute to a smoother optimization landscape, making the norm of full APG generally smaller than DPG, as illustrate on Figure This is not the case when the policy is conditioned on high-dimensional visual observations. As we will show shortly in Section when the sensor model g is involving complex rendering process, adding the regularization term B tends to increase the overall gradient norm, which hinder optimization.
Experimental validation A comparison between full APG and our DPG in full-state space provided in Appendix D.I while results under visual observations presented in Section 4
5

---

xee
Cartpole
X36
Hopper
X10
Ant
x10
Humanoid
7.5
Return
5.0
2.3
2.5
0.0
0
10
29
Time (hour)
Time (hour)
(hour)
(hour)
2203
x1o7
8.5 k1x0
Return
0.0
2.5
5.0
7.5
0.5 1. 15
45 55
Environment Steps (million)
Environment Steps (million)
Environment Steps (million)
Environment Steps (million)
Ours
DrQ v2
DreamerV3
Figure 4: Comparison with RI: our method achieves substantial speedups and significantly higher rewards across all tasks. Each curve shows the average performance over five random seeds, with shaded areas indicating standard deviation. In the humanoid task, dashed lines represent the final rewards attained by each algorithm at the end of training. The top row highlights wall-clock efficiency; the bottom row illustrates sample efficiency, with curves truncated the maximum number of simulation steps for better visualization.
3.2 Decoupled visual based analytical policy gradient
Here, we introduce Decoupled Visual-Basse Analytical Policy Gradient (D.Va), visual policy learning method built upon the decoupled policy gradient formulation. Our method is an on-policy algorithm that updates the policy using parallel simulation generate short-horizon trajectories. Following SHAC, rollouts resume from previous endpoints and reset at task termination. Trajectories discarded after each iteration reduce I/O overhead. To capture temporal cues such as velocity and acceleration, we stack three consecutive image frames-followig common practice in prior work [Hafner al. 2023 Mu al. 2025). We also provide ablation study on the number of stacked frames in Appendix D.3 The stacked frames are then encoded by a convolutional network to produce latent representation h, for the actor. The critic Ve in Equation (3) plays a crucial role in achieving good overall performance. (See Appendix D.2) For efficiency. we train the critic in the low-dimensional state space S as opposed to the observation space O. Although the critic is trained using state information, the policy remains aaatstggoottic throughout the entire training process, enabling direct deployment downstream tasks without requiring access to privileged state information. Full algorithm details are provided in Appendix B.1
B.1
Ours
0 08
4 Experiment
DrQ-2
1.34
We design our experiments to compare the proposed method against common visual policy learning algorithms GPU- accelerated simulation. Performance is evaluated based on final return, wall-clock time, and the number of environment steps. All hyperparameters are listed in Appendix C while additional details on setup are provided in the Appendix
DreamerV3
35.35
CURL
2.53
Percentage (%) Spent (hours)
Forward Simulation Planning
NN Fitting Time
B
Figure 5: Training time for Ant over 1M steps. Left:: phase percentages, where "planning" in DreamerV3 refers to roll- outs by learned world model. Right: ab- solute times used per 1M steps. Most time in visual RL is spent fitting neural networks.
4.1
Comparison to RL methods
Baseline
We compare our method with: (1) DrQ-v2 Yarats
al. 202 a model-free method combining image augmen-
lations with DDPG [Lillicrap et al., [2015]: (2) CURL [Laskin
al. 2020a model-free RL approach using contrastive learning and SAC (Haarnoja al. 2018); ; and (3) DreamerV3 [Hafner et al.) 2023). model-based algorithm that learns world model and uses for planning. All RL basclines are implemented with parallelized simulation take advantage of faster forward rollout.
Our approach achieves comparable sample efficiency to existing methods; however, excels in wall-clock time and final returns, as shown in Figure The discrepancy between sample and wall-clock time efficiency is because RI. methods reuse past experiences through replay buffers, whereas on-policy method discards samples after each iteration. Consequently, RL methods spend
Results

---

Cartpole
x3o1
Hopper
X12
Ant
x30
Humanold
7.5
Return
3.0
2.5
10
0.5
1.0
15
0
2
10
Time (minute)
Time (hour)
Time (hour)
Time (hour)
K30
XXp
X10*
X200
7.5
7.5
Return
5.0
2.5
2.5
3.0
7.5
10
20
20
40
20
60
Environment Steps (million)
Environment Steps (million)
Environnant Staps (million)
Environment Steps (million)
Ours
State-to-Visual Distillation Expert Policy
State-to-Visual Distillation Student Policy
Figure 6: Compare with State-to- Visual distillation: Our method matches distilled policy in final return and computation time on three simple tasks, but significantly outperforms it on the more challenging Humanoid locomotion task. The student policy's x-axis is shifted to account for expert training time. Our implementation builds on Mu al. 2025]. with enhanced expert training phase by replacing SAC with SHAC.
Cartpole
Hopper
Ant
Humanoid
Return
Time (minute)
Time (minute)
Time (hour)
Time (hour)
xIe
x10'
Return
2
10
15
Environment Stops (million)
Environment Stops (milllon))
Environment Stops (milllion)
Environment Stops (million)
Ours
SHAC with Differential Rendering
Figure 7: Compare to SHAC with differential Rendering: Our method matches SHAC on 2D tasks (Cartpole challenging 3D tasks (Ant and Humanoid).
and Hopper) and outperforms SHAC
more time updating neural networks-sun as fitting Q-functions DrQ-v2-than collecting new data. As illustrated in Figure 5] only a negligible proportion of time is spent on forward simulation for the RL baselines, indicating limited potential for speedup from faster simulators. In contrast, our method allocates a comparable amount of time both forward simulation and backward policy updates, suggesting it could further benefit from better simulation.
4.2
Comparison to method using privileged simulation
State-to-visual distillation Another class of popular methods [Loquercio et al. 2021 Chen et al.,
2023 training visual policies by first learning expert policy with privileged state access, then transferring knowledge to a visual policy via DAgger [Ross et al. 2011]. Among these, Mu et al. stopping when the behavior cloning loss low and using off-policy data from a replay buffer, which greatly reduces computation and improves performance. Our implementation of State-to- visuul distillation is based on the approach proposed by Mu et al. (2025). with one key modification: we use SHAC in place of SAC for expert training. We observe that SHAC consistently outperforms model-free SAC in settings where differentiable simulation available Xu al. (202T), resulting in further reduced computational time and improved final returns for training State-to- Visual distillation.
2025 proposes key design choices-eatt
Our method performs comparably to state-to-visua distillation three relatively easy tasks in terms of both computation time and final rewards. these tasks, both approaches achieve returns on par with the expert policy trained in the state space. However, on the more challenging
Results
7

---

Ours
Distillation
Figure 8: Our method learns policies that camera-aware, whereas the distilled policy from expert often adopts postures that are partially occluded or blocked from the camera view.
humanoid running task, our method significantly outperforms state-to-visual distillationacchieeeee high final returns, while distilled policy plateaus substantially lower values.
Key difference Here, we highlight the key difference between our method and state-to-vision distillation. While both can be viewed as behavior cloning from another policy, the source of the mimicked policy differs fundamentally. In state-to-visur distillation, actions imitated from frozen "expert" policy. In contrast, our method mimics actions from the "teacher" that provides incremental improvements to the current policy. We argue that learning from an incremental "teacher" may be more effective than learning from a fixed "expert", especially for complex tasks. First, expert actions may differ significantly from those produced by the current policy, making them harder to imitate accurately. Second, expert policy may provide ineffective corrective feedback in those state spaces that are visited by the current policy but rarely seen during its own training phase. In other words, the expert cannot handle situations for which it has no prior experience, We hypothesize that these two factors contribute to the performance gap observed on the humanoid task.
Another subtle but noteworthy difference is in the postures generated by the learned policies. As illustrated in Figure 8 we empirically find that method tends to produce more camera-aware behaviors, whereas the distilled policy often results in self-occluded postures. Although both the "expert" policy used in distillation and the "teacher" corrections in our method are agnostic camera views when providing supervision, the on-policy and iterative nature of our approach may lead to important differences. During the student policy distillation phase, imitating actions from unblocked visual inputs may be easier and converge quicker than learning from occluded views, which can introduce ambiguity. Since our method continually discards outdated rollouts and relies on recent data, the training process may be implicitly biased toward favoring trajectories that offer clearer, more informative perspectives.
4.3
Comparison to SHAC with differentiable rendering
Finally, we compare our method against ana- lytical policy gradient approaches that incorpo- rate differentiable rendering. To the best of our knowledge, no existing open-source baseline combines differentiable rendering with analyt- ical policy gradient method; therefore, we pro- vide our own implementation.
Hopper
Humanoid
30
Gradient Norms
200
2014
100
100
201
Ours
Rendering
We implement differentiable rendering using Py- Torch3D (Ravi al. [2020], in contrast to prior works [Wiedemann et al. [023 Liu et al. [2024), which rely on learned renderer. Using dif- ferentiable renderer based on computer graphics eliminates the need to optimize a separate neu- ral network and helps avoid compounding errors from distribution shifts as the scene evolves. provided in Appendix B.2 We then train the visual policy end-to-end under the SHAC framework. For fair comparison, we use identical simulations and neural architectures for both methods, and the value functions are all defined on low-dimensional state space.
Figure 9: Comparison of gradient norms between SHAC with differentiable rendering and V.VA: In this experiment, conducted in the visual space where sensor model g represents complex rendering process, the control regularization term adds to a noisy optimiza- tion landscape. As a result, SHAC generally exhibits a larger gradient norm compared to V.
Further details on our parallel rendering setup are

---

Results Figure 7 compares our method with SHAC across four benchmark problems. We find the performance of SHAC highly dependent on whether the task 2D or 3D. In the 2D tasks, i... Caripole and Hopper, SHAC achieves performance similar to ours. However, in 3D tasks, our method consistently outperforms SHAC, with SHAC failing to learn effective locomotion. We hypothesize that the discrepancy arises from the noisy optimization landscape introduced by the complex rendering process. In contrast to the low-dimensional state space-Where where the control regularization term B (5) acts residual connection and the norm of the full APG is generally smaller than that of DPG (see Figure3) the the high-dimensional visual space involves a more complex sensor model g. which includes 3D transformations and rasterization process. As result, the additive control regularization term contributes to a noisier optimization landscape. As shown in Figure the gradient norm of SHAC with differentiable rendering is generally langer than that of our method. Notably, for 3D tasks, the gradient norm in SHAC can rapidly exceed 10ts making the backward signal pure noise. In addition to smoother optimization, several other factors make method preferable SHAC for training visual policy in practice. First, our method is significantly more memory efficient, as shown in Figure T0 Second, scene complexity increases, i.e., Table I: Backward time for single training episode.
the number of meshes and the number of ver- tices per mesh, SHAC's memory usage grows rapidly. This is because the Jacobian with re- spect to each mesh vertex must be stored to
Hopper
Ant
Humanoid
Ours
0.11(s)
0.19(s)
0.21(s)
0.68(s)
SHAC
0.36(s)
0.51(s)
0.58(s)
1.38(s)
construct the computation graph. In contrast,
our method avoids storing these large Jacobians, resulting in a relatively stable memory footprint. Therefore, our method is more suitable for tasks involving a greater number of objects and higher- resolution meshes. Third, our method reduces the computational overhead during neural network updates by avoiding the multiplication of large Jacobian matrices associated with rendering during the backward pass. As shown Table []] our backward simulation 2-3x faster than SHAC, measured on the same machine. Lastly, developing high-quality differentiable rendering software demands substantial engineering effort. In contrast, our method does not depend on such software, enabling easier integration into existing simulation ecosystems [oooorvv et al., 2012 Xian et al.) 2024], with the potential to handle multi-modaa observation such as point clouds or LiDAR scans.
Related Work
Memory Usage (GB)
Durs
10.0
7.5
Visual policy learning Techniques integrating learned image en- coders [Finn et al., [2015] Mnih al.] 2015) have shown promising results
2.5
in visual policy learning. Building on these approaches, algorithms such as Kostrikov et al. [2020]. Yarats et al.
[2021] improve performance
Carppol Hopper Ant Humanoid
through data augmentation. Laskin et al. 2020a b), Stooke et al. [2021) leverage contrastive learning. and et al. 2019 2023), Hansen et al.) (2022) address the visual-control problem by learning world models for online planning. While these methods effectively tackle image-control challenges, they often require extensive environmental interaction and suffer from computational inefficiencies. In this work, we introduce a method that leverages differentiable simulation to reduce training time.
Figure 10: Peak mem- ory usage during train- ing. Our method is 3-5x more memory ef- ficient than SHAC, Each task is run with 32 paral- lel environments and short horizon length of 32.
Analytical policy gradient methods [Freeman et al. 2021 Qiao et al.] 2021 Mora et al., 2021] have gained traction with the rise of differentiable simula- tion. Among them, SHAC [Xu et al. 2021 mitigate noisy optimization via short-horizon rollouts and value function, making a core technique in many downstream robotics applications [Schwarke cct al. 2024]] Song et al.) 2024]. In subsequent work, Xing et al. 22025] further enhanced performance by adding entropy term to the optimization objective, smoothing the optimization landscape and
Policy learning with differentiable simulation
improving stability.
Recent efforts have adapted APG to visual policy learning: Wiedemann et al. 2023 incorporates differentiable rendering, Thile|Heeg et al. (2024) uses cropped vision features for quadrotor racing. Luo et al. [2024] propose hierarchical design separating control and perception. However, these approaches remain task-speeific and engineering-heavy. In contrast, our method generalizes across tasks and achieves higher computational efficiency than prior visual policy learning methods.

---

Limitation and Future
key limitation of our method is that requires both the system dynamics and the reward function to be differentiable. Differentiable dynamics often rely on smoothing or approximations, which can reduce modeling accuracy, while designing differentiable reward functions is itself nontrivial. An important direction for future work is to relax these requirements by integrating alternative optimization techniques, such as random search [Howell al., 2022 Liu al. [ or finite- difference methods [Mania et al. 2018
7 Conclusion
Our algorithm, D. Va, is a computationally efficient method for training visual policies utilizing differentiable simulation and first-order policy gradients. With this approach, we are able to train complex visual policies within hours using modest computational resources. We believe that this improvement in computational efficiency can unlock new possibilities for the robotics community, enabling practical end-to end-to-end training of policies from raw observations. However, this limitation shared by all baseline methods, training directly in the real world is often impractical. Future research should focus on how to effectively transfer the success of training visual policies in simulation to real-world scenarios.
Acknowledgments and Disclosure of Funding
This work is supported by the National Science Foundation under award NSF FRR 2238066. Any opinions, findings, and conclusions recommendations expressed in this material those of the authors and do not necessarily reflect the views of the National Science Foundation.
We additionally thank the reviewers for their valuable suggestions and Davis Zong for assisting with additional experiments during the paper revision.
References
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. mn: A vision-language- action flow model for general robot control, 2024. URL https: //app:://arxvv... orgooog//aabo abs/ /2100 24164
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End end learning for self-driving cars. arXiv preprint
2016.
Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8(84): eadc9244, 2023.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2024.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Learning visual feature spaces for robotic manipulation with deep spatial autoencoders. arXiv preprint 10X0::5509.0//11 13, 25(2), 2015.
C. Daniel Freeman, Erik Frey. Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax differentiable physics engine for large scale rigid body simulation, 2021. URL. http 7/g thub. www haamm/goggle/bbre
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. Pmlr. 2018.
10

---

Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565. PMLR, 2019.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint 2xXiv:3301.04104 2023.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.044: 04055, 2022.
Johannes Heeg. Yunlong Song, and Davide Scaramuzza. Learning quadrotor control from visual features using differentiable simulation. arXiv preprint arXiv:2410.15979 2024.
Taylor Howell, Nimrod Gileadi, Saran Tunyasuvunakool, Kevin Zakka, Tom Frez, and Yuval Tassa. Predictive sampling: Real-time behaviour synthesis with mujoco, 2022. URL https
//arxiv. 2211g
2212. 00541
Marco Hutter, Christian Gehring, Dominic Jud, Andreas Lauber, C Dario Bellicoso, Vassilios Tsounis, Jemin Hwangbo, Karen Bodie. Peter Fankhauser, Michael Bloesch, et al. Anymal-a highly mobile and dynamic quadrupedal robot. In 2016 IEEE/RS/ international conference on intelligent robots and systems (IROS), pages 38 38.44. IERE, 2016.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning drive in a day. In 2019 international conference on robotics and automation (ICRA), pages
IEEE, 2019.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint urXiv:2004. 1 13649, 2020.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. Proceedings of the 37th International Conference Machine Learning. Vienna, Austria, PMLR 1 119, 2020a. arXiv:2004.04136
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in neural information processing systems, 33: 19884-19895, 2020b.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0297) 2015.
Ruoshi Liu, Alper Canberk. Shuran Song, and Carl Vondrick. Differentiable robot rendering, 2024.
URL https://arxixi http://aaxvvvvogaaaaa/// www./bs//////eeeeooo http:/bbb/2411... 2410, 13851
Yilang Liu, Haoxiang You, and Ian Abraham. Hybrid mode sample-based control. In RSS 2025 Workshop on Whole-body Control and Bimanual Manipulation: Applications in Humanoids and Science and Systems (RSS), 2025a. Workshop paper.
Beyond. Robotics:
Yilang Liu, Haoxiang You, and Ian Abraham. Sample-basee based hybrid mode control: Asymptotically optimal switching of algorithmic and -indiffeerentiable control modes, 2025b. URL https
//arxiv.org .rrg//rggaa... 2510 19074
Antonio Loquercio, Elia Kaufmann, René Ranftl, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Learning high-speed flight in the wild. Science Robotics, 6(59))abbg5810 2021.
Jing Yuan Luo, Yunlong Song, Victor Klemm, Fan Shi, Davide Scaramuzza, and Marco Hutter. Residual policy learning for perceptive quadruped control using differentiable simulation, 2024. URL attps://arxiv.e wttp:///rgabb///a.../../ 03076
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive
approach to reinforcement learning, 2018. https://::///rr www..ogg///a..e.....o...oooe 0/0//88030000 07055
Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients are not all you need. arXiv preprint arXiv:2111.05800 2021.

---

Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):5295533 -33, 2015.
Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In International Conference on Machine Learning, pages 7805-7817. PMIR, 2021.
Tongzhou Mu, Zhaoyang Li, Stanistaw Strzelecki, Xiu Yuan, Yunchao Yao, Litian Liang, and Hao Su. When should we prefer state-lo-visual dagger over visual reinforcement learning? Proceedings of the AAAI Conference Artificial Intelligence, 2025.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming Cin. Efficient differentiable simulation of articulated bodies. In International Conference on Machine Learning, pages 8661-8671. PMLR, 2021.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Lo. Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv:2007.0885 088501 2020.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635 JMLR Workshop and Conference Proceedings, 2011.
Clemens Schwarke, Victor Klemm, Jesus Tordesillas, Jean-Pierre Sleiman, and Marco Hutter. Learn- ing quadrupedal locomotion via differentiable simulation. arXiv preprint arXiv:2404.02887 2024.
Yunlong Song, Sangbae Kim, and Davide Scaramuzza. Learning quadruped locomotion using differentiable simulation. arXiv preprint arXiv:2403. 148.44444 2024.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International conference on machine learning, pages 9870-9879. PMLR, 2021.
Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning, pages PMLR, 2022.
Richard S Sutton, Andrew G Barto, et al. Introduction reinforcement learning, vol. 135, 1998.
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Amav Gurha, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint vrXiv:2410.00425. 2024.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: 2099/RRSS.2012.6388
Nina Wiedemann, Valentin Wüest, Antonio Loquercio, Matthias Müller, Dario Floreano, and Davide Scaramuzza. Training efficient controllers via analytic policy gradient. In 2023 International Conference on Robolics and Automation (ICRA). IEEF, 2023.
Zhou Xian, Qiao Yiling, Xu Zhenjia, Wang Tsun-Hsuan, Chen Zhehuan, Zheng Juntian, Xiong Ziyan, Zhang Mingrui Wang Yian, Ma Pingchuan, Wang Yufei, and Dou Zhiyang. Genesis: universal and generative physics engine for roboties and beyond, December 2024. URL.
https://giie
com/ Coon//enesis--e Gemesis-Embodied Twweeed-AAAGGeeee AI/Genesis
Eliot Xing, Vernon Luk, and Jean Oh. Stabilizing reinforcement learning in differentiable multiphysics simulation. International Conference on Learning Representattinns (ICLR), 2025.
12

---

Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation, In International Conference on Learning Representations, 2021.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images, 2020. URL htttttttss/aiiii org/abs hrggabs/1910..o 01741
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning, arXiv preprint urXiv:2107.0966 2021.
13

---

Appendix
A Additional derivation
A.1 Derivative of analytical policy gradient separation
In this section, we provide detailed derivation of how the analytical policy gradient VaV [ can be decomposed into the decoupled policy gradient VoV (4) and control regularized term B We achieve this decomposition through pattern matching.
We begin with the decoupled policy gradient 4 which is obtained by taking the expectation over the gradient of individual trajectories. The gradient of each trajectory, in turn, is computed by summing the gradients of the temporal running rewards, as follows:
T
J (so. 0,8)
Σ Ver
(11)
1=
The gradient of the running reward, Ver,, is composed of two partial derivatives. First, the immediate reward rt is directly influenced by the action taken at time step t, denoted a,. This yields the partial derivative term:
Dr, Oa:
Da. 00
Second, the reward Tt also depends on the state sf, which itself depends on the previous state st 1 and action a;-1- In the decoupled formulation, this dependency propagates backward through time, leading to recursive gradient computation. Specifically, the second term is
Ôr, ds.
Üs: d0
where the derivative is given recursively by: de
de.
0s. dsi
+
Dar-1 30
Altogether, we
de
Ua
dB
Ua.
have
Vor, dr; 0a: +
dr. ds: as, d0
(12)
Da 00
Figure 72-
illustrates a typical backward flow decoupled policy gradient starting with the reward
Decoupled Pollcy Gradient
Control Regularization
Shared Policy Parameters
Reward
Forward
Backward gradient start with r2. Left: the backward gradient flow on decoupled policy gradient. Right: the backward flow on control regularization. Each subplot in the row starts from the reward node r2 and traces backward along the computation graph until it reaches the policy parameters O, illustrating partial contribution the total derivative.
Figure 11:
We examine how the control regularization term 5 is formulated. Following the structure illustrated in Figure I]] the control regularization is constructed by summing a series of terms of the form:
Ôr, Oa, do, ds;
da, do; ds, d0
14

---

Here, the total derivative de, ds:
can be computed recursively as follows:
de
da.-1 :ai-1 do;-1 ds:- Os: ds:-1
+
(13)
de
a
00 do,-1 ds d0
ds d0
Altogether, the control regularization term is given in Eq. ( Thus, the full analytical policy gradient the sum of the decoupled policy gradient and the control regularization term:
VoV = VoV+
(14)
B
Additional implementation details
B.1
D.VA algorithm
Our critic training follows SHAC, minimizing the mean squared error collected trajectories:
Critic Learning
Lo B
[IVo(s) V(s)
(15)
where
h-1-1
((x) = (1-
Σ
G + x
(16)
is the estimated value function and is treated as constant target during critic learning. Here, G (Eko0> R(S )(+))) + +VVV2(( represents the k-step return from time I, and Var
delayed critic function used stabilize the training process [Mnih et al. 2015].
Full algorithm Below, we summarize our Decoupled Visual-Based Analytical Policy Gradient VA) algorithm.
Algorithm 1: DA (Decoupled Visual Based Analytical Policy Gradient)
Function Rollout ()
Initialize N initial states 8n-
for 1=
do with torch. no _)_gddd))
compute pixel images o, = g(r)..
Sample actions a, (o,)), simulate and compute rewards and next states S1+11 end for
Collect N trajectories {(St, ae, re)1 and compute actor loss Lo via Eq.
Return т, Le
Function Main()
Initialize ma, Vib Vgr +- Vo M do
for epoch
Generate N short-horizon trajectories 7 and compute actor loss L. by calling Rollout (). Compute decoupled policy gradient V and update Ta with Adam.
Fit value function Va via critic loss (15) and update delayed target Ver aV (1 V).
end for
B.2 Differentiable Rendering
In this section, present the construction of our differentiable renderer, designed facilitate end-to-end training.
Given a state vector s = lq qTTT, where q E R" represents joint positions and q denotes joint velocities, which have dimension m 1 or m depending on the presence of quaternions. We compute m homogeneous transformations T for forward kinematics. These transformations encode the rotation and translation from the world frame to each joint frame and are used to transform geometry meshes defined in local joint frames to assemble the full robot mesh. Lighting and camera poses are also updated using these transformations to set up the full scene. Finally, we render the scene using Dorch33D via rasterization. Figure 12 summarizes the whole process.
15

---

9
St = [
Camera & Lighting
Assembly
Rasterization
Forward Kinematics
Meshes
Scene
Rendered Image
Forward
Backward
Figure 12: Diagram for differentiable rendering
C Hyperparameters
In this section, we describe hyperparameter tuning process and share additional insights gained during experimentation. Detailed hyperparameter values are provided at the end.
C.1 How we tune hyperparameters
For all baseline methods, we initialized hyperparameters using values reported in the original papers. We then identified the most sensitive hyperparameters and tuned them sequentially. Additional attention is given to those hyperparameters emphasized in the original works. This process involved over 500 experiments, with some trials manually terminated early when it became evident that the chosen hyperparameters
suboptimal. In contrast, did not apply additional tuning to proposed method, as the combination hyperparameters from state-based SHAC and the encoder architecture from DrQV2 already yielded strong performance.
summarize our key findings from the tuning process and present the final hyperparameter values used in the following sections.
C.2 Key findings and experimental settings
DrQ-v2 We implement DrQ-v2 with parallelized forward simulation improve wall-clock time efficiency. During each training episode, we concurrently collect new samples using the current policy across multiple parallel environments. After data collection, we perform several update steps on Q-functions using the data from the replay buffer. The forward pass accounts for only small portion of the total training time with parallelization, as shown in Figure 5 The majority of computation time is instead dedicated to updating the agent's 's neural network, i.e., Q-function and Actor. We found that the ratio between agent update steps and new sample collection plays a critical role in achieving good performance. While performing multiple agent updates per environment step improve sample efficiency, it may negatively impact wall-clock time performance. In addition, we observe that excessive updates harm final policy performance due outdated data from the replay buffer. Therefore, we carefully tune both the number of parallel environments and the number of updates per training episode, aiming maximize wall-clock efficiency and final return. The hyperparameters used in our experiments are listed in Table 2 The primary difference from the original DrQ-v2 setup that we increase both the number of parallel environments and the number of agent updates per step, while the ratio is kept the same. Unless otherwise noted, the same parameters are used across all four tasks. We kept neural architecture identical to that used the original paper and summarize in Table 3 and
CURL We also apply parallelized forward simulation to CURL to accelerate training. Similar to DrQ-v2, the update frequency plays critical role in achieving strong performance. We tune the number of parallel environments to optimize wall-clock efficiency within the available GPU memory budget. The training parameters used for CURL are listed in Table 5 Notably, CURL uses a similar
16

---

Table 2: DrQ-v2 training parameters
Parameter name
Value
Number of parallel environments
32: 16 for Caripole
Number of agent updates
16; 8 for Cartpole
Replay buffer capacity
10"
Action repeat
2
Mini-batch size
256
N-step returns
3
Discount factor 7
0.99
Learning rate
10 10- (8 10 for Humanoid)
Critic Q-function soft-update rate
0.01
Exploration stddev. clip
0.3
Exploration stddev. schedule
linear .0, 0.1, 500000);
linear(1.0, 0.1, 2000000) for humanoid
Table 3: DrQV2 encoder architectures
Parameter name
Value
Input image size (height x width) 84 x 84
Convolution kernel size
3, 3, 3, 3
Convolution output channel size
32, 32, 32, 32
Convolution activation function
Relu
Table 4: DrQ-v2 actor-critic architecture
Task name
Trunk size
Policy network
Critic network
Cartpole
50
[1024, 1024]
[1024, 1024]
Hopper
50
[1024, 1024]
[1024, 1024]
Ant
[1024, 1024]
[1024, 1024]
Humanoid
100
[1024, 1024]
[1024, 1024]
architecture to DrQ-v2 (see Table 3 and 4) both are adopted from Yarats et al. [2020]. To ensure fair comparison, the same architecture for both algorithms, avoiding confounding factors introduced by architectural differences.
DreamerV3 Our implementation of DreamerV3 builds upon the open-source repository available We parallelized the environment stepping in our implementation; however, we observed that this parallelization has minimal impact on per- formance-consistee with the findings reported by the author of the repository. Additionally, DreamerV3 is memory-intensive algorithm, which limits the degree parallelism we can apply. The detailed hyperparameters for training are mostly kept the same those used in Hafner et al. 2023 and listed in Table 6 The neural architecture remains unchanged from the Hafner et al. 2023
at htttps://githur com/ NM5 h0m.12/reamerv3-toor
D.Va (Ours) Architecture details: Stacked images are first processed by an encoder to generate hidden state. The encoder is 4-layer convolutional network, identical that used DrQ- v2 (Table B). The hidden state is then passed to the actor network to generate actions. The actor network consists of trunk network and policy network, following the design DrQ-v2. The trunk network is single linear layer followed by layer normalization. The policy and critic network is adopted from the state-based SHAC: ML. network with EL. activation and layer normalization. The detailed network architectures are provided in Table
17

---

Table 5: CURL training parameters
Parameter name
Value
Number of parallel environments
16
Number of agent updates
8
Replay buffer capacity
10
Action repeat
2
Batch size
32
Discount factor Y
0.99
Actor learning rate
10
Critic learning rate
10-3 3
Adam (31, for actor and critic
(0.9, 0.99)
Q-function soft-update rate
0.01
Initial temperature
0.1
Temperature learning rate
10-1
Adam (31,32) for temperature
(0.5, 0.99)
Table 6: DreamerV3 training parameters
Parameter name
Value
Image size (height x width)
(64 x 64)
Number of parallel environments
Batch size
16
Batch length
64
Train ratio
512
Action repeat
2
Replay buffer capacity
106
Action repeat
2
Discount factor
0.997
Discount lambda
0.95
Actor learning rate
x 10 55
Critic learning rate
x 10
Actor-critic adam epsilon
10-6
World model learning rate
10
World model adam epsilon
10-8 8
Critic EMA decay
0.98
Reconstruction loss scale
1.0
Dynamics loss scale
0.5
Representation loss scale
0.1
Actor entropy scale
3 x 10
Return normalization decay
0.99
Table 7: D. Va actor-critic architecture
Task name
Trunk size
Policy network
Critic network
Cartpole
64
[64, 64]
[64, 64]
Hopper
128
[128, 64, 32]
664, 64]
Ant
128
[128, 64, 32]
[64, 64]
Humanoid
256
[256, 128]
[128, 128]
For training, we apply a linear decay schedule to adjust the learning rate over episodes, with specific hyperparameters provided in Table g
18

---

Table 8: D.Va training parameters
Parameter name
Cartpole
Hopper
Ant
Humanoid
Short horizon length h
32
Number of parallel environments N
64
Actor learning rate
0.002
Critic learning rate
0.0002
0.002
0.0005
Target value network a
0.2
0.995
Discount factor y
0.99
Value estimation
0.95
Adam (B1, (32)
(0.7, 0.95)
Number of critic training iterations
16
Number of critic training minibatches
State-to-visual Distillation The architecture is kept identical to that used for D. Va, which can be found in Table and 7 As described earlier, the architecture constructed by simply concatenating the DrQv2 encoder with the SHAC state-based architecture, ideally. no method is unfairly favored. The hyperparameters are tuned following the guidelines of Mu et al. 2025]. We found that, to make Patte-tooVisual Distillation work effectively, the most critical factor is the data collection strategy and the frequency of network updates, consistent with the findings reported Mu al.) (2025]. Specifically, we collect data in SHAC-style manner: instead of executing long, continuous trajectories, we roll out short-horizon segments that resume from the endpoint of the previous rollout. Notably, this implementation is the same as that used in Mu et al. 2025 The detail parameters are listed T
Table 9: State-to-visual Distillation training parameters
Parameter name
Value
Short horizon length h
32
Number of parallel environments N
Learning rate
0.002
Adam (81, 92)
(0.7, 0.95)
Batch size
128
Harly stop threshold
0.1
Maximum reply buffer size
10"
SHAC with differentiable rendering The neural architecture is kept identical to ours and is detailed in Table 3 and Table 7 For 3D tasks such as Ant and Humanoid, we observed that the gradient norm quickly diverges infinity as the horizon length increases. To address this, we use an even smaller horizon length compared to the one used for training state-based SHAC. The number of parallel environments also reduced due to GPU memory constraints. The specific parameter values are provided in Table T0
D
Additional Experiment
D.1
Compare DPG to SHAC on state space
We provide additional experiments to valid using decoupled policy gradient is enough to gain good performance in many scenarios. Here, we conduct on state space, with a single line change on the original SHAC code: actions actor (oba) to actions actor .ebacc ()). We kept all hyperparameters identical to those reported SHAC. Figure F compares the training performance of the SHAC with full analytical policy gradient T to our decoupled policy gradient 4 We find that
19

---

Table 10: Visual-SHAC training parameters
Parameter name
Cartpole
Hopper
Ant
Humanoid
Short horizon length h
32
32
32
Number of parallel environments N
64
64
32
32
Critic learning rate
0.0002
0.002
0.0005
Target value network a
0.2
0.995
Actor learning rate
0.002
Discount factor
0.99
Value estimation
0.95
Adam (B1, /92)
(0.7, 0.95)
Number of critic training iterations
16
Number of critic training minibatches
X12
Cartpole
X10
Hopper
X10>
X10*
Humanold
5.0
Return
0.0
0.0
Environment Steps (million)
Environment Steps (million)
Environment Steps (million)
Environment Steps (million)
Short Horizon Actor Critic
Decoupled Policy Gradient
Figure 13: Comparison between DPG and SHAC on state space: Experiments
conducted with low-
dimensional observations and results are averaged five random seeds. All hyperparameters are kept identical those used in original SHAC. DPG still achieves comparable results in the settings that favor SHAC.
the decoupled policy gradient achieves performance comparable to SHAC on low-dimensional state spaces, even under settings that favor SHAC.
D.2 Ablation Value Function
In this section, we present an ablation study the value function. As shown in Figure 14 without the value function, our method fails to learn an effective policy. This result highlights the criticall role of the value function in analytical policy gradient methods.
Cartpole
Hopper
Ant
Humanoid
7.5
Return
0.0
75
0.5
Environment Steps (million)
Environment Steps (million)
Environment Stops (million)
D.Va Value Environment Steps (millionn)
w/ Value Function
Figure 14: Ablation study value function. D.Va without value function fail learn effective control for most of the tasks.
D.3 Ablation on number frames
In this section, we present an ablation study on the effect of the number of concatenated frames used as input to the policy. Table [1 shows the final return achieved with different numbers of stacked frames as input to the policy. We observe that as long as the number of frames is not extremely low (c.g., one), the method achieves comparable final performance.
20

---

Table 11: Final Return Achieved with Different Number of Stacked Frames
Number of frames
Caripole
Hopper
Ant
Humanoid
2068 = 29.83
4398.48 + 266.80
5681 = 1344.52
5596.94 = 937.92
2
2115 = 21.20
5055.23 = 6.73
7418 + 862.39
5342.00 = 1680.70
3
2139 + 22.42
5067.13 18.14
7218 + 986.00
7475.77 812,48
4
2155 + 25.49
5055.03 = 2.08
9680 + 222.27
7498.79 #498.79-633.11
5
2095 = 22.5
5072.33 14.62
8286 880.50
6345.61 37.76
6
2140 = 29.84
5077.63 = 23.74
8243 + 799.45
6578.59 = 717.24
D.4
Additional tasks
We include an additional quadruped walking task using ANYmal [Hutter al. 2016]. with hiir--patty side images as imput. As shown in Figure [5 our method learns a visual locomotion policy within 20 minutes on a single GPU.
Figure 15: ANYmal locomotion: Our method is able to learn ANYmal locomotion from purely third-party visual input within 20 minutes.
D.5 More gradient norm analysis
Figure shows the gradient norms of the visual policy computed using SHAC and D. Va, respectively, We observe that the gradient norms for the two 3D tasks blow up, which explains why SHAC with differentiable rendering fails to learn an effective visual policy for these tasks, as shown Figure
Cartpole
Hopper
Ant
Humanoid
2014
Gradlent Norms
15
200
1631
10
100
1014
10
1000
1500
2000
4000
150
200
Returns
Returns
Returns
Returns
Ours
SHAC with Differential Rendering
Figure 16: Gradient norms computed using SHAC and D. D.Va are shown for all environments. The gradient norms for the Ant and Humanoid tasks blow up.
E
Setup
E.1
Tasks Descriptions
We select four classical RL tasks across different complexity levels. The camera views are similar to those used in Yarats et al. [002]] illustrated in Figure 17 Except for Cartpole, where the camera fixed to the world frame, all other cameras track the position of the robot's 's base joint. The motion of Cartpole and Hopper is constrained to 2D, whereas Ant and Humanoid are free to move in 3D space. The body of the Ant may block the view of some its legs due to the side-view camera setup, making the environment partially observable. In contrast, the joints in all other environments remain visible from the camera regardless of posture, resulting in fully observable settings.
The reward functions are identical to those used in the SHAC paper, except for the Cartpole system, where we add health score prevent the cart from moving off-screen. The details are summarized as follows:
21

---

Cartpole
Hopper
Ant
Humanoid
Figure 17: Camera views each environment ManiskilIV3
Cartpole:
The running rewards defined
R:
0² 0.10² 0.05x 0.1.²
(17)
where 0, x denote the angle of pole from upright position and position of cart; and 0, & the angular and linear velocity respectively. The total trajectory length is 240, and early termination is triggered when Cartpole is outside the camera views, i.c./ 2.5.
Hopper:
The running rewards are defined as
0
R:
Ur 300 + Reeight 0.1 0|1||a||,
(18)
where is the forward velocity, 0 is the orientation of base joint and
2000 A A
h1 An 50
Rooight
An clip(h + -1,
(19)
is designed to penalize the low height state. The total trajectory length is 1000, and early termination triggered when height the hopper is lower than -0.45m.
Ant: The running rewards are defined as
R: = Ur + 0.1 + Rieading + Ps 0.27,
(20)
where v, is forward velocity, Rap. Recading is the projection of base orientation in upright and forward direction, encouraging the agent to be vertically stable and run straightforward, p3 is the height the base. The total trajectory length is 1000, and early termination is triggered when the height of the ant lower than 0.27m.
Humanoid: The running rewards are defined
R: = Ur + 0.1 + Rheeding + Rneight 0.002||a]].
(21)
where U2 is forward velocity, Rup. Pieeading is the projection of base orientation in the upright and forward direction, and
-200 As 50 An. A > 0
Recight
An = clip(h - 0.84, -1,011.
(22)
The total trajectory length is 1000, and early termination is triggered when the height of the torso is lower than 0.74m.
E.2 Simulation
We use the same differentiable simulation framework proposed in SHAC the underlying dy- namics model. For the three benchmark RL methods and the state-to-visual tasks, we employ ManiSkill-V3 Tao al. 2024) for rendering. contrast, for visual-SHAC, we implement a cus- tom differentiable rendering pipeline using PyTorch3D, as detailed in Appendix B.2 All software components are GPU-accelerated and parallelized. We evaluate our method under both rendering pipelines. To ensure fair comparison, all experiments presented in Section 4 are conducted using
22

---

the same rendering setup across different methods. We find the ManiSkill implementation to be fairly efficient approxiattlly 3 faster than our differentiable rendering pipeline- and therefore, use differentiable rendering only when necessary. However, aside from the difference in forward rendering speed, we find that the final return and sample efficiency of method remain similar across both rendering pipelines.
E.3 Hardware
All experiments are conducted on single NVIDIA GeForee RTX 4080 GPU (16GB) with an Intel Xcon W5-2445 CPU and 256GB RAM. Unlike the case of simulating dynamics alone-where tens of thousands of environments can be parallelized at once- once- heterogeneous rendering requires significantly more memory. As result, our hyperparameter tuning is carefully constrained to stay within the available memory budget.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarize our contribution into three parts:
Claim: We provide novel computationally efficient method for training visual policy.
Support: The detailed implementation of our algorithm is provided in Section 3 and
Appendix [.1] The experimental validation is provided Section !
Claim:
We benchmark common visual-policy learning methods under GPU-accelerated
simulation.
Support: The experiments provided in Section 4 Additionally, provide computational bottleneck analysis for selected methods.
Claim: We provide an analysis of analytical policy gradients and show deep connection between open-loop trajectory optimization with closed-loop trajectory optimization Support: The analysis is provided in Section 3 with both conceptual explanations and numerical evidence.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The main limitation of our method lies in its reliance on the quality of the simulation environment, discussed in Section 7 However, we argue this limitation is shared by most existing methods in the field. As result, our next goal to transfer the success to real-world scenario.
Justification:
The computational efficient of our method across different complexity is demonstrate in Section 4
The scaling of memory usage is also shown in Figure
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
24

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Theorem | follows directly from straightforward application of the chain rule and does not require any assumptions beyond differentiability.
Justification: We provide a formal proof of Theorem I in Section 3
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All additional information to reproduce the results including the general setting
and hyperparameters are includes in Appendix B Appendix C and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
25

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: the code is available
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c ec/
gablic/guidess CodeSubmissionoo///
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. www.c//wwwpgpcuuuccccoepooooppee http://gdesssseeeee www.oo/ewwiiiiisoooooooiooeo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Appendix C and Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
26

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For each task and algorithm, we run experiments with five different random seeds and report both the mean and standard deviation in Section following common practice in the reinforcement learning literature.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:: httpts/se/eerrrsss..pppccc//ceee
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics. Our work adheres fully to these guidelines and does not involve any specific social concerns or ethical risks.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
27

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not explicitly discuss societal impacts the paper. However, our work may contribute to reducing the carbon footprint by enabling more efficient algorithms. We do not foresee any potential negative societal impacts arising from this research, as focused on foundational algorithmic improvements without direct application to sensitive or high-risk domains.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not involve any models datasets that pose high risk of misuse. Therefore, no additional safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
28

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our implementation entirely based on open-source software and code. We properly cite all tools and packages used in work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets released as part of this work the time of submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: this paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
29

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
30


Reviews:
[{"review_id": "XG2coHhyuc", "reviewer": "Reviewer_GuqR", "summary": "This paper introduces D.Va, a novel method for training visual control policies efficiently with first-order analytical policy gradient using differentiable simulation.\nThe core innovation, the decoupling of visual observations from the computation graph during BP enables more efficient training.\nThe authors prove the effectiveness of their method with extensively benchmarking diverse approaches and theoretical analysis.", "questions": "See in weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## strengths\n1. The decoupling visual policy learning method in the paper is novel. And the authors provide sufficient formal and experimental analysis to prove the effectiveness of the DPG(Decoupling Policy Gradient).\n2. Extentive comparisons on performance, wall-clock time and memory across diverse methods are made. And the performance gains well support the method.\n3. The paper is well-strucutured, and figures are clear.\n\n## weaknesses\n1. In this work, the authors use stacked frames (3 frames) to provide temporal cues. In many tasks in locomotion like running, 3 frames may be enough. How sensitive is the method to the number of frames? \n2. While decoupled policy gradient has shown great performance, but as analyzed in the paper, the term of control regularization reflects how past trajectory influence current decision making. So the decoupling strategy might not be so effective on long-horizon dependent tasks.\n3. Also, dropping the term of control regularization makes the policy optimization to be blind towards observation. So the reviewer suspects that the decoupling strategy might succeed easily on single task, and wonders the performance of its generalization ability.\n4. $V_{\\phi}$ plays a crucial role in achieving good overall performance. But actually this could only work for sim-to-real paradigm. This can also explain why the DPG is stronger than SHAC with differentiable rendering on 3D tasks, as it has access to the privileged states.", "was_revised": true, "final_justification": "This is a technically solid paper, which could also lead to many exicting future research directions. It also provides many detailed experiment results and statistics analysis. Readers can easily follow and reproduce the results. My concern have been all well addressed after the rebuttal.", "rebuttal": "# Author’s response to reviewer GuqR\n\nWe sincerely thank reviewer GuqR for their valuable feedback on our paper and their positive comments on the novelty of the decoupling visual policy learning method.\n\nWe address the reviewer’s concerns as follows: \n\n**Q1: In this work, the authors use stacked frames (3 frames) to provide temporal cues. In many tasks in locomotion like running, 3 frames may be enough. How sensitive is the method to the number of frames?**\n\n> **A1**: We conducted additional experiments to evaluate the sensitivity of our method to the number of stacked frames and report the final returns achieved below.  In short, we observe that as long as the number of frames is not extremely low (e.g., 1), the method achieves similar final performance, which corresponds to results in [1]\n\n> ### Table: Final Return Achieved with Different Number of Stacked Frames\n>\n> | Environment | Frame 1          | Frame 2           | Frame 3           | Frame 4           | Frame 5           | Frame 6           |\n> |-------------|------------------|-------------------|-------------------|-------------------|-------------------|-------------------|\n> | Hopper      | 4398.48 ± 266.80 | 5055.23 ± 6.73    | 5067.13 ± 18.14   | 5055.03 ± 2.08    | 5072.33 ± 14.62   | 5077.63 ± 23.74   |\n> | Humanoid    | 5596.94 ± 937.92 | 5342.00 ± 1680.70 | 7475.77 ± 812.48  | 7498.79 ± 623.11  | 6345.61 ± 37.76   | 6578.59 ± 717.24  |\n\n\n**Q2: The value function plays a crucial role in achieving good overall performance. But actually this could only work for sim-to-real paradigm. This can also explain why the DPG is stronger than SHAC with differentiable rendering on 3D tasks, as it has access to the privileged states.**\n\n> **A2** :Thank you for the comment. We’d like to clarify that both DVA and SHAC use privileged state information to train the value function (as noted in Lines 255–256), so the performance gap is not due to unequal access to privileged information.\n\n> The key difference lies in the optimization approach: DVA uses a surrogate decoupled gradient that leads to more stable training (which is supported by Figure 9 and discussion in Lines 266-268). As shown in Figure 9, SHAC often produces extremely large gradients (up to $10^{15}$), leading to high-variance training, while DVA maintains more smaller and regular gradient norms, leading to less variance.\n\n**Q3: While decoupled policy gradient has shown great performance, but as analyzed in the paper, the term of control regularization reflects how past trajectory influence current decision making. So the decoupling strategy might not be so effective on long-horizon dependent tasks.**\n\n> **A3**: Thank you for the insight! We agree with Reviewer GuqR that our method alone may face challenges in long-horizon planning tasks and could result in sub-optimal trajectories. However, it is unclear whether this limitation is due to the removal of the control regularization term or due to SHAC itself, which already uses a short planning horizon. Analytical policy gradient methods like ours are in general better suited for high-frequency (e.g. 50Hz) controls that involve agile motions. Many downstream applications—such as those in [2, 3, 4]—require visual feedback and quick reactions, where our method can be especially effective. To better handle long-horizon dependencies, it is still possible to integrate our approach into a hierarchical RL framework, where a high-level planner guides a low-level controller—an approach shown to be effective in prior work such as [5]. We see this as a promising direction for future research. \n\n**Q4: Also, dropping the term of control regularization makes the policy optimization to be blind towards observation. So the reviewer suspects that the decoupling strategy might succeed easily on single task, and wonders the performance of its generalization ability.**\n\n> **A4**: Thank you for raising this important point. We acknowledge that removing the control regularization term may reduce the coupling between the policy and the observation, potentially making the policy less sensitive to camera pose. We briefly mention this concern in Lines 231–233 of our paper.\n\n> However, this challenge is not unique to our method—it also arises in widely used techniques like teacher-student distillation (our second class of baselines), which are commonly applied in downstream visual-policy training for robotics tasks [1, 2, 3]. Interestingly, we observe that our method exhibits a degree of camera-awareness in practice: the learned policy often biases locomotion toward more favorable viewing angles—more so than the teacher-student distillation baselines (see Figure 8). While this is an empirical observation, it suggests that decoupling does not eliminate all sensitivity to visual input.\n\n> That said, we agree that generalization under partial observability remains a major challenge, and our current method alone may not fully address such cases. As partial observability is a long-standing issue in RL, a full treatment is beyond the current scope of this work, but we appreciate the reviewer’s feedback in helping us clarify this discussion. We see this as a promising direction for future work. \n\n**References**:\n\n[1] Shang, Wenling, et al. \"Reinforcement learning with latent flow.\" Advances in Neural Information Processing Systems 34 (2021): 22171-22183.\n\n[2] Zhuang, Ziwen, et al. \"Robot parkour learning.\" arXiv preprint arXiv:2309.05665 (2023).\n\n[3] Liu, Minghuan, et al. \"Visual whole-body control for legged loco-manipulation.\" arXiv preprint arXiv:2403.16967 (2024).\n\n[4] Antonio Loquercio, et al.” Learning high-speed flight in the wild”. Science Robotics, 6(59):eabg5810, 2021.\n\n[5] Cheng An-Chieh, et al. \"Navila: Legged robot vision-language-action model for navigation.\" arXiv preprint arXiv:2412.04453 (2024)."}, {"review_id": "PDkhgZno4T", "reviewer": "Reviewer_vvNV", "summary": "The authors propose to study mechanisms for efficient policy learning in visual tasks. They take advantage of differentiable simulators to compute an approximation for the full gradient that that not differentiate through rendering terms. They find this is a strong speed-up over baselines.", "questions": "Given how fast this trains, why not run more experiments on more environments? Normally I wouldn't ask such a question but these seem like fairly light compute environments and it would allow you to actually make the claims you do in the conclusion like \"\"\n- Do the results in Figure 9 hold in the other 2 environment used on the paper?\n- The claim on line 172 that the value function is critical for performance. Where is the experiment supporting that claim?\n- I understand that there are analyses in this, but methods-wise, what is different here than just running SHAC and not differentiating through the observation function? If it's just that, should it really have such a distinct name?\n- GIven how large the gradient norms are, why is gradient clipping not needed here? Or is it used?", "limitations": "This paper does not really address weaknesses and limitations of the method. The main limitation explicitly discussed, reliance on simulators, is not really a limitation of the method. It is hard for me to tell the authors what the limitations of their methods are, but a more substantive discussion would have improved the paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- the baselines seem to be carefully tuned and explanations are provided for the tuning procedures\n- the sample complexity seems to have serious improvements over the baselines\n- seeming improvements in wall-clock speed over the original method it is derived from\n\n### Weaknesses\n- addressed in questions\n\n\n###  Suggestions (which you can ignore)\n- This paper would be more convincing if it was run on more environments. Two of the environments, Cartpole/Hopper, are fairly straightforward to solve with any method. Seeing this work across many environments might induce people to use it.", "was_revised": true, "final_justification": "My questions primarily concerned some lack of justification for some of the claims and an insufficient discussion of the limitations. The authors have addressed them and so I'm keeping my already high score.", "rebuttal": "# Author’s response to reviewer vvNV\n\nWe sincerely thank reviewer vvNV for their valuable feedback on our paper and their positive comments on the effectiveness of our method. \n\nWe address the reviewer’s concerns as follows: \n\n**Q1: Given how fast this trains, why not run more experiments on more environments? This paper would be more convincing if it was run on more environments. Two of the environments, Cartpole/Hopper, are fairly straightforward to solve with any method. Seeing this work across many environments might induce people to use it.**\n\n> **A1**: We appreciate the reviewer’s suggestion to include additional environments, as broader validation is indeed valuable. In quick response, we have already added experiments on the Anymal locomotion task with side-view cameras—a more realistic robotic control scenario during the rebuttal period. Our method successfully learns effective locomotion policies in under 15 minutes. \n\n> While our initial submission focused on carefully benchmarking and tuning state-of-the-art methods—which required substantial computational resources—we recognize the importance of demonstrating versatility. After the rebuttal period, we plan to expand our experiments to a broader set of robotics-oriented tasks and include more examples in the final version.\n\n**Q2:  Do the results in Figure 9 hold in the other 2 environment used on the paper?**\n\n> **A2**: Yes, and we will include figures for other environments in our final version.\n\n**Q3 The claim on line 172 that the value function is critical for performance. Where is the experiment supporting that claim?**\n\n> **A3**: Thank you for the suggestion! We’ll include an ablation study in the final version to highlight the role of the value function. In the meantime, we’ve added a table showing the final return of DVA with and without the value function. More broadly, value functions are known to significantly aid training in analytical policy gradient methods—not only in visual tasks (see, e.g., Figure 4 in SHAC [1]).\n\n> ### Table: Final Returns Achieved With and Without Value Function\n| Environment | Without Value Function | With Value Function     |\n|-------------|------------------------|--------------------------|\n| Hopper      | 454.33 ± 49.34         | 5067.13 ± 18.14          |\n| Humanoid    | 539.56 ± 15.52         | 7475.77 ± 812.48         |\n\n**Q4: I understand that there are analyses in this, but methods-wise, what is different here than just running SHAC and not differentiating through the observation function? If it's just that, should it really have such a distinct name?**\n> **A4**: Yes, the key difference from SHAC is that we stop the gradient through the observation (rendering) function. However, we argue that this change leads to meaningful improvements—especially on visual tasks by stabilizing training (See Figure 9).\n\n> We gave it a distinct name to highlight the impact of this decoupling, similar to how methods like DrQ [2] or RLPD [3] build on existing algorithms with simple yet effective modifications. We hope that DVA can serve as a reference point for applying this strategy within analytical policy gradient methods.\n\n**Q5: Given how large the gradient norms are, why is gradient clipping not needed here? Or is it used?**\n> **A5**: Yes, we apply gradient norm clipping to both our method (DVA) and SHAC during training. However, simply clipping the norm proves insufficient when gradients become as extremely large as the $10^{15}$ values seen with SHAC in our Humanoid experiments (Figure 9). The ineffectiveness of analytical policy gradients in the presence of such extreme values has been extensively studied in previous work (e.g., [4, 5]).\n\n**Q6: This paper does not really address weaknesses and limitations of the method. The main limitation explicitly discussed, reliance on simulators, is not really a limitation of the method. It is hard for me to tell the authors what the limitations of their methods are, but a more substantive discussion would have improved the paper.**\n> **A6**: Thank you for pointing this out—we appreciate the opportunity to reflect more deeply on the limitations of our method. We agree that the reliance on simulators is a broader challenge shared by many reinforcement learning approaches, and may not uniquely characterize our method. We included it because, in practice, simulator quality often becomes a key bottleneck when transferring methods to real-world scenarios.\n\n> More specifically to our approach, one limitation is that it builds upon analytical policy gradient methods and therefore requires full differentiability of the trajectory—this includes not just the dynamics, but also the reward function. While differentiable rewards can be naturally defined for many tasks (e.g., locomotion, manipulation) it becomes more challenging for tasks with sparse or binary feedback (e.g., success/failure). Designing differentiable proxies for such rewards may require additional heuristics. We will update the paper to include a more detailed discussion of these limitations and thank the reviewer again for the helpful suggestion.\n\n**References**:\n\n[1] Jie Xu, et al., “Accelerated Policy Learning with Parallel Differentiable Simulation,” ICLR 2021.\n\n[2] Denis Yarats, et al., “Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.”  ICLR. 2021.\n\n[3] Ball Philip J., et al., “Efficient online reinforcement learning with offline data.” ICML, 2023.\n\n[4] Luke Metz, et al. “Gradients are not all you need”. arXiv preprint arXiv:2111.05803, 2021.\n\n[5] Hyung Ju Suh, et al., “Do differentiable simulators give better policy gradients?” ICML. PMLR, 2022."}, {"review_id": "zma0iIC3b5", "reviewer": "Reviewer_cpR5", "summary": "This paper presents D.VA, a novel on-policy AC algorithm for visual policy learning in differentiable simulation. It proposes to decouple the visual part gradients and builds a two-stage optimization method, which can be interpreted as an open-loop trajectory optimization and distillation. The method is evaluated on four classic RL environments, demonstrating better wall-clock time and even outperforming off-policy algorithms in sample efficiency. The authors also provide the profiling results and a comparison with teacher-student distillation with state-based teacher policy, which is very informative.", "questions": "- Can you explain in details why the incremental teacher is more easily to mimic than the expert frozen teacher in lines 222 to 224?\n- Based on the two-step optimization approach, it is similar to the idea of DIPO and DDiffPG that first optimize the action respect to the Q-gradient and then imitate the optimized action. Therefore, can author explain the connection with those off-policy algorithms and is it possible to extend the current method into an off-policy fashion?\n- In Fig. 3, the grad norm of DPG is much larger than APG, however, the authors claim that the training of DPG is actually more stable in line 39. Is there evidence to support this claim as I didn't find anything in the method or experiments?", "limitations": "Limitation is discussed in the conclusion.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- the idea is novel and well-motivated\n- the paper is generally well-written and easy to follow\n- provides comprehensive and diverse experiments to give better understanding of the method\n\nWeakness:\n- the authors claim that the decoupled policy gradients provides smooth optimization and stable training. However, there is no evidence to support and in Fig. 3 the gradient norm is even larger than the counterpart, which I think larger gradient usually leads to unstable learning.\n- the tasks are relatively simple. In line 137 the authors mentioned the impact of dropped bias term, which captures the interdependence between the actions. Therefore it is interesting to explore how the proposed method works in more challenging tasks, such as long-horizon tasks.", "was_revised": true, "final_justification": "The authors provided a comprehensive evaluation to support claims made in the manuscript. In the rebuttal, the authors further discussed the potential future directions and limitations in details, making the paper more solid.", "rebuttal": "# Author’s response to reviewer cpR5\n\nWe sincerely thank reviewer cpR5 for their valuable feedback on our paper and their positive comments on the novelty of our method and the comprehensiveness of our experiments.We address the reviewer’s concerns as follows: \n\n**Q1: the authors claim that the decoupled policy gradients provides smooth optimization and stable training. However, there is no evidence to support and in Fig. 3 the gradient norm is even larger than the counterpart, which I think larger gradient usually leads to unstable learning.\nIn Fig. 3, the grad norm of DPG is much larger than APG, however, the authors claim that the training of DPG is actually more stable in line 39. Is there evidence to support this claim as I didn't find anything in the method or experiments?**\n\n> **A1**: Thank you for the question. Figure 9 provides empirical support that our decoupled policy gradient improves training stability—showing much smaller gradient norms than SHAC. In high-dimensional visual tasks SHAC gradients can explode (up to $10^{15}$), stalling learning. In contrast, our gradients stay well-behaved and enable smoother optimization.\n\n> As for Figure 3, it's a low-dimensional proof-of-concept (as noted in Lines 157–162) meant to validate that our surrogate provides a descent direction—not to claim improved stability. While the gradient norm there is higher than SHAC, both are within a manageable range (below $10^2$), and standard practices like gradient clipping are effective, as supported by our results (Appendix Figure 13) and prior work [1, 2].\n\n**Q2: The tasks are relatively simple.**\n\n> **A2**: We evaluate our method across a range of classical RL environments with varying levels of difficulty. Notably, on the challenging visual humanoid locomotion task, our approach achieves up to 4× higher return compared to baselines (see Figure 4,6,7), while also demonstrating the fastest wall-clock training time.\n\n> In addition, we are working on adding experiments on more complex robotics-oriented tasks, such as the Anymal robot running task, which features more complex dynamics and partial observability due to large body occlusions. Our method is able to learn effective locomotion policies on Anymal in under 15 minutes, further highlighting its efficiency and adaptability.\n\n**Q3: Can you explain in details why the incremental teacher is more easily to mimic than the expert frozen teacher in lines 222 to 224?**\n\n> **A3**: Thank you for the question. There are two main reasons why an incremental teacher (online) is easier to mimic than a frozen expert (offline). First, as noted in Line 224, a frozen expert can produce actions that differ significantly from the student policy. This naturally raises a crucial implementation choice: how close to mimic newly collected data. Mimicking too closely slows down training due to heavy supervised updates, while mimicking too loosely can degrade data quality generated in future cycles and detract from the final performance (as the student collects data under its own imperfect policy, the expert’s training distribution is no longer align well with the states visited by the student). Following best practices from [4], we extensively tuned crucial hyperparameters such as update ratios and added tricks like early stopping to get strong performance. In contrast, our method avoids this trade-off by directly applying gradient ascent, simplifying training and improving stability.\n\n> Second, as mentioned in Lines 225–228, a fixed expert doesn’t adapt to state distribution visited by students. The incremental teacher, however, co-evolves with the student, providing more on-policy guidance and resulting in a higher final performance gain.\n\n**Q4: In line 137 the authors mentioned the impact of dropped bias term, which captures the interdependence between the actions. Therefore it is interesting to explore how the proposed method works in more challenging tasks, such as long-horizon tasks.**\n\n> **A4**: Thank you for highlighting this point. We agree that it is an interesting and important direction to explore how our method performs in long-horizon settings. However, our current approach is built on top of SHAC, which already truncates the planning horizon to a short window. This makes it difficult to isolate the effect of dropping the bias term in the context of long-horizon tasks. Meanwhile, extending our method within a hierarchical RL framework—where a high-level planner provides guidance in subgoal, may help address long-horizon dependencies, as demonstrated in prior work [3]. We view this as a promising direction for future research.\n\n**Q5: Based on the two-step optimization approach, it is similar to the idea of DIPO and DDiffPG that first optimize the action respect to the Q-gradient and then imitate the optimized action. Therefore, can author explain the connection with those off-policy algorithms and is it possible to extend the current method into an off-policy fashion?**\n\n> **A5**: Thank you for pointing us to DIPO and DDiffPG. We were not aware of these works during the course of our project, and we find them highly relevant and inspiring to us. Below we address the reviewer’s questions based on our understanding of these two works:\n\n> **1. Connection to off-policy methods like DIPO and DDiffPG**:\nOur method is fundamentally on-policy, where state-action pairs are discarded after each training episode. In that sense, our approach is conceptually closer to DDiffPG than DIPO. While we present our method as a two-step optimization (trajectory optimization followed by supervised policy update) for the sake of conceptually understanding decoupled policy gradient, in practice we directly compute the decoupled policy gradient and apply gradient ascent rather than explicitly generating new trajectories and behavior cloning as done in DIPO/DDiffPG. This design removes the need to manually tune the learning rate for the trajectory optimization step, which can be sensitive. As noted in Equation (7), our method supports arbitrarily small learning rates in the trajectory optimization phrase, allowing it to handle highly non-smooth trajectories. The difficulty of optimizing such nonsmooth trajectories is absorbed into the behavioral cloning phrase (Equation 8), which is generally easier to optimize in practice. This unified treatment automatically balances optimization difficulty across the two stages.\n\n> **2. Extending to off-policy learning**:\nYes, our approach can potentially be extended to an off-policy setting. We are actively exploring this in our concurrent work. A natural way to do so is to divide decoupled-policy gradient into the two stages—trajectory optimization and behaviour—and apply multiple updates to each phrase using existing data. However, we find there are many challenges in this setting, such as choosing a suitable learning rate for trajectory optimization on non-smooth objectives and maintaining a suitable ratio between new data collection and policy updates. Extending our method to off-policy learning is definitely important direction and we are currently investigating.\n\n**References**: \n\n[1] Cheng, An-Chieh, et al. “Navila: Legged robot vision-language-action model for navigation.”, arXiv preprint arXiv:2412.04453 (2024).\n\n[2] Hyung Ju Suh, et al. “Do differentiable simulators give better policy gradients?”, ICML, 2022.\n\n[3] Cheng, An-Chieh, et al. \"Navila: Legged robot vision-language-action model for navigation.\", arXiv preprint arXiv:2412.04453 ,2024.\n\n[4] Tongzhou Mu, et al.. “When should we prefer state-to-visual dagger over visual reinforcement learning?”, AAAI, 2025."}, {"review_id": "Ik4mLfLf7V", "reviewer": "Reviewer_YuQp", "summary": "The paper studies the problem of optimizing visual policies using first-order gradients with differentiable simulators. One major problem is that computing first-order gradients in high-dimensional image spaces is computationally expensive. To address this, the paper proposes decoupled policy gradient (DPG) that avoids back-propagation through the rendering process. The paper proves that DPG is identical to open-loop trajectory optimization. Finally, the paper integrates the proposed method with SHAC, resulting in decoupled visual-based analytic policy gradient (D.Va). The final proposed algorithm achieves superior performance compared to model-free and model-based RL algorithms, including DrQ-v2, DreamerV3, and SHAC with differential rendering.", "questions": "- Could you also compare your method with online RL algorithms that leverage privileged information, such as [1]?\n- Is there a particular reason you chose not to provide privileged information to the critic?\n- Why did you select DreamerV3 rather than TD-MPC2 [2] as the model-based RL baseline? Since your experiments use the ManiSkill-V3 simulator, where TD-MPC2 has already been tested, TD-MPC2 would be the more appropriate choice.\n\n[1] Edward S. Hu, et al., Privileged Sensing Scaffolds Reinforcement Learning, ICLR 2024 \\\n[2] Nicklas Hansen, et al., TD-MPC2: Scalable, Robust World Models for Continuous Control, ICLR 2024", "limitations": "The authors have adequately addressed the limitations of the proposed method in the conclusion section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- (Originality) Extension of first-order gradient policy optimization from low-dimensional state spaces to high-dimensional image spaces is novel.\n- (Quality) The proposed method is simple yet effective.\n- (Clarity) The gradient analysis in Figure 3 is very helpful for understanding the effectiveness of the proposed method.\n- (Quality) The paper compares the proposed method with many different baselines, including zeroth-order RL algorithms and first-order policy gradient algorithms.\n\nWeaknesses\n- (Quality) The authors should have fairly compared their proposed method with RL with privileged information.\n- (Significance) I am not sure that the proposed method is better than first training an expert policy using state information only (which is 5~10x more efficient as shown in Figure 13) and then distilling it to an image-based policy (which is also efficient as shown in Figure 6).", "was_revised": true, "final_justification": "The proposed method is simple yet effective, and the presentation is clear. While it would be better to include more baselines that use privileged information, the current results are already sufficiently strong. All of my concerns have been addressed. Accordingly, I am raising my score to 5.", "rebuttal": "# Author’s response to reviewer YuQp\n\nWe sincerely thank reviewer YuQp for their valuable feedback on our paper and their positive comments on the novelty and overall good performance of our method. We address the reviewer’s concerns as follows: \n\n**Q1: I am not sure that the proposed method is better than first training an expert policy using state information only (which is 5~10x more efficient as shown in Figure 13) and then distilling it to an image-based policy (which is also efficient as shown in Figure 6).**\n\n> **A1**: We argue that the wall-clock time for student-teacher distillation methods should be measured end-to-end—that is, from training the state-based teacher to obtaining the final visual policy. Our method eliminates the two-stage process and trains the visual policy directly from scratch. This one-stage training offers a key advantage: learning begins immediately, whereas in student-teacher pipelines, visual policy training only starts after the teacher is fully trained. In our experiments, our method often reaches near-final performance before distillation in the two-stage baseline even begins.\n\n> In addition to training time, another important aspect is the final returns that can be achieved. As shown in Figure 6, our method continues to improve in the visual humanoid locomotion task, ultimately achieving returns comparable to expert policies trained with privileged state input. In contrast, the two-stage methods plateau much earlier, with significantly lower final performance.\n\n**Q2:  Is there a particular reason you chose not to provide privileged information to the critic? The authors should have fairly compared their proposed method with RL with privileged information.**\n\n> **A2**: We do use privileged state information to train the critic, as noted in Lines 172–174 and 254–256. Additionally, for a fair comparison, all baselines in Sections 4.2 (state-to-visual distillation) and 4.3 (SHAC + differentiable rendering) also incorporate privileged state information during training.\n\n**Q3: Could you also compare your method with online RL algorithms that leverage privileged information, such as [1]?**\n\n>**A3**: Thank you for pointing us to [1]. While [1] focuses on more challenging POMDP settings, our experiments are conducted on standard visual RL benchmarks that are closer to fully observable environments (see Figure 14). In this case, adding additional sensor modalities may not lead to significant performance improvements. \n\n> However, we especially appreciate the reviewer to introduce [1] to us and the associated S3 benchmark —it looks like a strong testbed for evaluating methods like ours under partial observability. This also aligns with Reviewer GuqR’s feedback and highlights an exciting direction for future work. Based on [1]’s taxonomy, our method could be seen as using both a privileged critic and a privileged world model (via differentiable simulation), and we’re looking forward to exploring this further.\n\n**Q4: Why did you select DreamerV3 rather than TD-MPC2 [2] as the model-based RL baseline? Since your experiments use the ManiSkill-V3 simulator, where TD-MPC2 has already been tested, TD MPC2 would be the more appropriate choice.**\n\n>**A4**: We initially considered both TD-MPC2 and DreamerV3 as baselines. However, in our experiments on visual MuJoCo benchmarks—especially the more challenging humanoid tasks—we observed that TD-MPC2 failed to learn effective locomotion policies. In contrast, DreamerV3 performed more robustly and achieved significantly better results across visual tasks.\n\n> Additionally, as noted in the original TD-MPC2 paper, their method is not specifically designed for visual RL tasks. Their reported results only include simple 2D visual task environments and do not report more complex visual locomotion tasks as evaluated in our paper. Based on both empirical performance and intended design scope, we chose DreamerV3 as a more suitable and competitive model-based RL baseline for our visual control experiments.\n\n> ### Table: Final Returns Achieved for Humanoid Tasks with Visual Inputs\n | Method       | Final Return    |\n|--------------|----------------------------|\n| Ours         | 7475.77 ± 812.48           |\n| Dreamer-V3   | 1517.00 ± 571.63           |\n| TD-MPC2      | 50.70 ± 10.52              |\n\n**References:**\n\n[1] Edward S. Hu, et al., “Privileged Sensing Scaffolds Reinforcement Learning”, ICLR 2024\n\n[2] Nicklas Hansen, et al., “TD-MPC2: Scalable, Robust World Models for Continuous Control”, ICLR 2024"}]

Author rebuttal:

Meta-review: This paper proposes D.Va, a computationally efficient method for visual policy learning that leverages differentiable simulation by decoupling the rendering process from the computation graph. The reviewers are in unanimous agreement that this paper should be accepted (all providing a score of 5).

The proposed D.Va is novel and simple, but also fast, with significant improvements in wall-clock training time and final performance when compared to a set of strong baselines. The rebuttal effectively addressed concerns about generalization, scope of the evaluation, and stability of the gradients. Overall, this is a technically solid paper with a clear and impactful contribution to visual policy learning.
Decision: Accept (spotlight)