Does “Do Differentiable Simulators Give Better Policy Gradients?” Give Better Policy Gradients?
Authors: Ku Onoda
Paavo Parmas
Manato Yaguchi
Yutaka Matsuo
Venue: ICLR 2026 Poster (2026)
Keywords: Differentiable simulation
Reinforcement learning
Policy gradient
Model-based reinforcement learning
Monte Carlo gradient estimation
Reparameterization gradient
Likelihood ratio gradient
Score function gradient estimator
Inverse variance weighting
Randomized smoothing
Primary area: reinforcement learning
TL;DR: Gradient estimators for policy learning with differentiable simulators that handle discontinuities robustly and remain stable in practice with simple variance control.

Abstract: In policy gradient reinforcement learning, access to a differentiable model enables 1st-order gradient estimation that accelerates learning compared to relying solely on derivative-free 0th-order estimators. However, discontinuous dynamics cause bias and undermine the effectiveness of 1st-order estimators. Prior work addressed this bias by constructing a confidence interval around the REINFORCE 0th-order gradient estimator and using these bounds to detect discontinuities. However, the REINFORCE estimator is notoriously noisy, and we find that this method requires task-specific hyperparameter tuning and has low sample efficiency. This paper asks whether such bias is the primary obstacle and what minimal fixes suffice. First, we re-examine standard discontinuous settings from prior work and introduce DDCG, a lightweight test that switches estimators in nonsmooth regions; with a single hyperparameter, DDCG achieves robust performance and remains reliable with small samples. Second, on differentiable robotics control tasks, we present IVW-H, a per-step inverse-variance implementation that stabilizes variance without explicit discontinuity detection and yields strong results. Together, these findings indicate that while estimator switching improves robustness in controlled studies, careful variance control often dominates in practical deployments.

Published as a conference paper at ICLR 2026
DOES "Do DIFFERENTIABLE SIMULATORS GIVE BET- TER POLICY GRADIENTS?" GIVE BETTER POLICY GRADIENTS?
Onoda Paavo Parmas Manato Yaguchi
Yutaka Matsuo
The University of Tokyo. Jokyo,Japan [ku. onoda, paavo. parmas, manato.. Manato.yaguchl. mat @weslab. t. ..u-tokyo.ac. jp.jp
ABSTRACT
In policy gradient reinforcement learning, access differentiable model enables Ist-order gradient estimation that accelerates learning compared to relying solely derivative-free Oth-order estimators. However, discontinuous dynamics cause bias and undermine the effectiveness of Ist-order estimators. Prior work addressed this bias by constructing a confidence interval around the REINFORCE Oth-order gradient estimator and using these bounds to detect discontinuities. However, the REINFORCE estimator is notoriously noisy, and we find that this method requires task-specific hyperparameter tuning and has low sample efficiency. This paper asks whether such bias the primary obstacle and what minimal fixes suffice. First, we re-examine standard discontinuous settings from prior work and introduce DDCG, a lightweight test that switches estimators in nonsmooth regions; with a single hyperparameter, DDCG achieves robust performance and remains reliable with small samples. Second, on differentiable robotics control tasks, we present IVW-H, per-step twerse-variance implementation that stabilizes variance without explicit discontinuity detection and yields strong results. Together, these findings indicate that while estimator switching improves robustness in controlled studies, careful variance control often dominates in practical deployments.
INTRODUCTION
Policy gradient methods seek to optimize a parameterized policy 0 by estimating the gradient of the expected return, 9 d Ep(T) [R(T)] the most general setting-where the environment is treated as black box-Oth-order estimators such as REINFORCE (Williams, 1992) are often used. While broadly applicable, these estimators suffer from high variance and poor sample efficiency. When a differentiable simulator is available, Ist-order gradient estimators (c.g., via the reparameterization trick (Kingma et al., 2015)) can substantially reduce variance and accelerate convergence. However, real- world systems often involve contacts, friction, other nonsmooth effects, producing discontinuities that bias Ist-order estimates (Lee et al., 2018; Parmas and Sugiyama, 2021).
Each approach has its own advantages and disadvantages, and one way leverage their strengths is by mixing the estimators (Parmas et al., 2018; 2023). Parmas et al. motivate this composite view by showing that chaotic dynamics can gradient variance explode with horizon, and advocate inverse variance weighting (IVW) as a principled way to set the mixture weights. Specifically, these methods compute
= 0و + (1 ))
where Åo and ğ are Oth- and Ist-order estimates respectively, and a E [0, 1] is a weighting parameter. Vidal Under IVW, a = Vi.o++V1.... When the variances are estimated accurately. IVW can improve performance by reducing variance. Despite its appeal, IVW may fail in domains with discontinuities or contact dynamics. As the work of Suh et al. (2022) shows, sharp changes the reward landscape create situations where the Ist-order gradient exhibits large errors but spuriously shows low empirical variance in finite samples. This phenomenon, called "empirical bias," " leads IVW to overweight corrupt 1st-order estimates, harming performance. Building the same a-mixing scheme above, Suh

---

Published as a conference paper at ICLR 2026
et al. (2022) propose alpha-order batched gradient (AoBG), which augments IVW with a discontinuity- detection rule based on confidence intervals around the REINFORCE estimator. However, since REINFORCE can be extremely noisy, these intervals are broad, reducing sample efficiency and necessitating extensive task-specific parameter tuning. Moreover, while prior reports discuss AoBG behavior, they do not establish robust success standard robotic control benchmarks (Gao et al., 2024).
In this paper, we pursue two concrete goals focused on reassessing composite gradient methods and examining whether the variance or empirical bias is the main obstacle to practical performance.
First, we reassess existing work and expose fundamental shortcomings. Specifically, we re-establish the existence of the simite-sample bias phenomenon-wheet 1st-order gradients can appear low- variance yet be inaccurate-and introduce Discontinuity Detection Composite Gradient (DDCG), which uses a lightweight statistical test to decide when to trust I st--rrder information. We reproduce and re-evaluate all experiments from the AoBG paper under the same settings (Suh et al., 2022) and show that DDCG achieves results comparable to better than AoBG, with substantially improved robustness to hyperparameters and reliable behavior ssall-sample regimes.
Second, we ask whether this bias is actually the primary obstacle in practical robotics control. Prior studies (Son al., 2023; Gao et al., 2024) reported limited performance or incomplete realizations of inverse-variance mixing; therefore provide clear, per-time-step implementation, IVW-H, to isolate the role of variance control in practice. On standard robotics tasks, IVW-H attains strong performance without explicit discontinuity detection, suggesting that stabilizing variance the step level can be sufficient, while the role of empirical bias appears minimal in these settings.
2 RELATED WORK
Differentiable Simulators. Recent advances differentiable simulators enable gradient-based policy optimization with either automatic differentiation (Griewank and Walther, 2003; Heiden al., 2021; Freeman et al., 2021) or analytic derivatives (Carpentier and Mansard, 2018; Geilinger et al., 2020; Werling et al., 2021). These methods reduce variance in gradient estimates and often accelerate learning. However, contact-rich or discontinuous dynamics remain challenging because the inherent nonsmoothness introduces bias or instability in Ist-order gradient estimates, undermining their reliability for optimization tasks.
Composite Gradient Estimators. Combining Oth-order and stt-order gradients can balance ro- bustness and efficiency. Parmas et al. (2018) propose Total Propagation (TP), which uses inverse variance weighting (IVW) to mix gradients. However, discontinuities can introduce biased Ist-order gradients (Lee et al., 2018; Parmas and Sugiyama, 2021), and IVW can fail when these biases underestimated. Suh et al. (2022) address this "empirical bias" phenomenon by scheme that constructs confidence intervals around Oth-order gradient estimates to detect bias.
Policy Optimization with Differentiable Simulation. Analytic Policy Gradient (APG) (Freeman et al., 2021) computes policy gradients directly from simulator-provided derivatives, accelerating learning but not explicitly addressing discontinuities. Short-Horizon Actor-Critic (SHAC) (Xu et al., 2022) reduces variance by truncating rollouts and using a terminal value to smooth the objective, enabling effective use of analytic gradients. Adaptive-Gradient Policy Optimization (AGPO) (Gao et al., 2024) mitigates nonsmoothness by adapting weights based on batch- grrehggradient variance, while downweights unreliable analytic gradients within PPO framework.
PPO (GIPPO) (Son et al., 2023) introduces an a-policy that
BACKGROUND
Notation. Throughout this paper, we use bold font (c.g., x) to represent tensors unless otherwise stated. Here, È denotes the sample mean of the corresponding quantity. We define the empirical variance of of N samples
N Σ
(),, - B1:) 2
VH
N - 1=1
2

---

Published as a conference paper at ICLR 2026
Task setting. We consider finite horizon control tasks with state variables and actions that
are computed from a policy πç. States transition according to the dynamics p(s'|s ss, a); following
actions according to the policy me leads to trajectories "C (So. dg, .1...... .(1) We consider the
objective IE [[T]]], where R(T() is a cumulative sum of scalar rewards computed by the reward function r(s, a). We aim to maximize this objective using gradient ascent.
central theme in estimating gradients or any statistical inference is the interplay between bias and variance. For an estimator 2 of Z, the mean squared error (MSE) can be expressed as
Bias- Variance Error Decomposition.
(2 2)
[ Z) + E 22 B[2))
(I)
Error
Bias
Variance
An estimator is unbiased if R[2] Z. In gradient-based methods, a low-bias estimator may still exhibit high variance, hindering learning efficiency. Conversely, reducing variance may introduce systematic bias. Balancing bias and variance is therefore a key challenge in designing gradient estimators, motivating strategies to control variance without incurring significant bias.
Elementary Gradient Estimators. We perform randomized smoothing and sample policy pa- rameters Ç ~ P({; 0). Let 0 denote the parameters to be optimized, and let Tę represent a random trajectory or episode whose distribution depends on A. In particular, in the current work p(C: 0) will always be Gaussian, with 0 as the mean of this Gaussian. That is, we can write
-= 0 + GE, 6 C~N(0,
A gradient estimator is unbiased if
d
El
Bp(r.:0) [R(Tc)]-
do
Here, "unbiased" refers to the estimator being unbiased for the gradient of the objective inside the simulation model (Eq. 3). Transfer errors from simulation the real world a separate concern addressed by sim-to-real techniques.
Oth-order estimator. A widely used unbiased method is the score function likelihood ratio approach (Glynn, 1990), often referred to as REINFORCE (Williams, 1992). It can be written as
(0) dlogp(T: 0) (R(T)-b)
d0
where 7 represents a sample from p(:; 0), and b is a baseline that can reduce variance (Berahas et al., 2022). In our experiments, follow Suh et al. (2022) and use deterministic baseline given by the objective evaluated at the mean parameter, b = ʃ(0)- Ex(r,,0) the Monte Carlo samples used in the gradient estimate and thus keeps the estimator unbiased. Despite being unbiased, this estimator often suffers from high variance, which can significantly increase the number of samples required for effective learning.
which does not depend on
ist-order estimator. An alternative approach, known as the reparameterization trick (Kingma al., 2015) or pathwise derivative (Schulman al., 2015), avoids directly differentiating through probability distribution by defining a deterministic transformation
T To(E),
p(E).
Because T still has distribution p(T; 0) by construction, one obtains the Ist-order estimator:
dR dTo(E)
9 (0) = dr d0
(6)
This estimator remains unbiased if R is continuous, and practice, often exhibits lower variance than 9o. Consequently, it tends to be more sample-efficient for continuous parameter and action aç spaces, For instance, if we reparameterize Ç as in Eq. (2), then a0 = I and öe which simplifies 9 to dR with respect to 0. However, when R is discontinuous, the 1 1st-order estimator can be biased.
Batch Estimation. While Eq. (4) and Eq. (6) are presented single-sample estimators for clarity, in practice (and in our experiments), these are estimated using a batch of N samples to increase accuracy, and compute empirical means and variances.
3

---

Published as a conference paper at ICLR 2026
1.0
10P
10-4-
Signoole
10
0.6
10
02
m
Eenor
Empiriccal Varinnce
T-0.01
Goadientt
0.0
Teraperature
(a) Discontinuous like behavior
(b) Error, Empirical Variance
Figure 1: Sigmoid Function
Composite Gradient Estimators. Although the 1st-order estimator 인 typically has lower variance than the Oth-order 9o. may be biased in the presence of discontinuities. A practical approach by Parmas et al. (2018) 18) mixes these estimators via linear combination:
f. = og + (1 az)go. E [0,
where a close to 1 emphasizes the 1 st-order estimator while a near O relies more on the Oth-order method. Additionally, they propose leveraging Inverse Variance Weighting (IVW) to optimally select in their Total Propagation (TP) framework. Under the simplifying assumption that go and 91 are uncorrelated, the theoretically optimal weight Copt that minimizes the variance go is
V [90]
Clopt
[
91
(8)
If the covariance between Åo and タ1 is non-negligible the above expression must be adjusted accordingly, as discussed in (Parmas al., 2023). Nevertheless, in the uncorrelated case,
ax
[91
indicating that the combined estimator can achieve strictly lower variance than either go or ğ alone.
In practice, the true variances V [ğo] and V [ف1] are generally unknown and must be approximated from sample data. Explicitly, one computes V V[gol and V(gn to obtain éupt. This creates difficulties whenever the empirical variance estimates are poor, notably discontinuous environments.
Limitations of Empirical Variance Estimation While IVW often performs well, Suh et al. (2022) points out that certain practical factors-such as contact, friction, or discontinuities in physics simulations-can cause an "empirical bias" phenomenon, resulting in gradients that exhibit low empirical variance yet remain highly inaccurate. An illustrative example involves the Sigmoid function, (ig
[+oxp(-*) As shown in Figure 1a, when the temperature T is large,
the function is fairly smooth. However, very small T, it transitions sharply and resembles discontinuity. Although Sigmoid(z) is mathematically continuous for any finite T,, its narrow transition region makes sinite-sample gradient estimates prone large, sporadic errors.
From the perspective of the bias-variance decomposition Eq. (1), an unbiased estimator's error coincides exactly with its variance (since Bias 0). In principle, this means that the true variance of the Sigmoid gradient should match the observed error. However, as Figure 1b shows, the empirical variance computed from small batch often fails to reflect the true error. The reason that very large gradients occur with small probability, causing the true variance to be very large (sometimes viewed as "infinite variance" in the limit of vanishing probability). A mathematical example illustrating how this "infinite variance" phenomenon arises is given in Appendix B. In practice, a finite sample may overlook those rare but significant gradients, leading to a systematic underestimation of the variance. This phenomenon underscores a fundamental challenge: when unbiased gradient estimator has heavy-tailed or rare maggiidde events, the empirical variance can severely underestimate the true variance.
Interpolation Protocol (AoBG) The AoBG method proposed by Suh et al. (2022) builds upon the IVW framework by introducing additional safeguards against discontinuities. AoBG starts with Oopi but modifies it based on a measure of potential bias B = | -o 20
Ckopt
if Olgg B < E,
02
(10)
otherwise.
4

---

Published as a conference paper at ICLR 2026
This formulation introduces precision threshold y to control acceptable bias and confidence term e to account for uncertainty in the Oth-order estimator. When potential bias is too large, the method reduces to maintain precision, effectively reverting to the reliable Oth-order estimator in challenging areas. For small sample sizes, a conservative approach uses only the Oth-order gradient (a = 0), though this raises several concerns.
First, with small sample sizes, the Ist-order estimator is typically more effective due to its lower variance. Thus, relying on the Oth-order gradient seems counterintuitive, potentially leading subop- timal outcomes. Second, selecting the parameter 7 for each task requires task-specific tuning, limiting the method's generalizability and usability. Eliminating the need for such parameter adjustments would make the method more robust and practical across diverse scenarios.
4 PROPOSAL
4.1 DISCONTINUITY DETECTION COMPOSITE GRADIENT (DDCG)
We propose Discontimuity Detection Composite Gradient (DDCG), which keeps the usual inverse- variance mix of Oth- and Ist-order estimators but gates the use of the 1st-order term by a simple statistical test. The gate derived from two standard conditions under which IVW trustworthy:
(A1) Reliable variance: the empirical variance of the Ist-order gradient is close to its true variance (so IVW weights are meaningful).
(A2) Local smoothness: / is locally well-behaved (e.g., near-quadratic), making the 1st-order gradient accurate and low-variance (Xu et al., 2019; Domke, 2019).
If (A1) holds, IVW already downweights noisy Ist-order terms; but (A2) is also needed to avoid using biased Ist-order estimates near discontinuities. We therefore run statistical test that passes with probability at least 1 when (Al) and (A2) hold; if it passes we apply IVW, otherwise we fall back to the Oth-order estimator. Importantly, these assumptions are not required for the algorithm to run: they are only checked to decide whether to trust IVW.
Step 1: Variance Reliability The first (AI) concerns the accuracy of the empirical variance estimate of Ist-order gradients. If this assumption holds, we can rely on the sample-based variance used by IVW to be close the true variance.
Formally, suppose we have N samples { from a function f. along with their function values (f(z))) and gradients (V/()))) Denote:
N
ú
Σ (V(..)
(11)
N
where V Vf=
N EN = fa;) is the empirical mean of the gradients. We assume that differs from the true variance of V ((x) by at most e.:
i Ex f E,, (VIVV(
(12)
Such bound can be derived via standard statistical results (e.g., chi-squared-basse confidence intervals). By enforcing a maximal floor on é, we reduce the risk of underestimating gradient variance, and thus overweighting potentially high-variance Ist-order estimator.
Step 2: Discontinuity Detection To derive the statistical test, we assume that f is sufficiently smooth so that Ist-order gradients remain accurate. In practice, smoothness ensures that the variance of sst-rrder estimates does not explode.
To merge (AI) and (A2) into single test, we assume a Lipschitz-like condition on gradient changes:
fV((x) f((iil ~ L|| 3/1.
(13)
where is a local curvature constant. We then compare the variance of a quadratic approximation of f with the empirical gradient variance. Under smoothness and bounded residuals, a condition emerges (detailed in Appendix C):
V[f(z) + Ev 2 2(1 - 0² 2 ?
(14)
5

---

Published as a conference paper at ICLR 2026
Here the right-hand side corresoonds to the gradient variance that a locally quadratic model of f would induce under randomized smoothing, while the term involving ||V/|| subtracts the contribution of the mean gradient. Intuitively. if f is smooth and our variance estimates are reliable, the empirical gradient variance cannot be much smaller than this quadratic proxy: large fluctuations in function values necessarily imply non-trivial fluctuations in the gradient. When the left-hand side falls below the right-hand side, interpret this as evidence of heavy- or discontinuous behavior that makes the IVW weights unreliable, and we fall back to the Oth-order estimator.
In implementation, all quantities in Eq. (14) are computed from the same finite batch of samples: replace (a)] by its empirical counterpart f=))= Analogously to Eq. (12), one could also attach an explicit confidence interval to this scalar variance estimate; we found that doing so did not materially change the decisions of the test, for simplicity we omit this extra term in the main algorithm.
Interpretation of c. In Eq. (14), the parameter e relaxes the requirement that f be perfectly quadratic. If were exactly quadratic, then taking с O would make the inequality tight in that ideal case. As c increases above we allow more deviation of f from perfect quadratic behavior, permitting greater nonlinearity or mild discontinuities. Thus, smaller imposes stricter smoothness requirements on f. while a larger c offers more flexibility for f to deviate from a purely quadratic shape. In our experiments simply fix = 0.3 across all AoBG benchmarks, and Appendix H shows that DDCG robust for any e [0.1,0.5 0.9] on all considered tasks.
Step 3: Adaptive Weighting Given the test in Eq. (14), we define the composite gradient estimator by adaptively selecting weight a between Oth- and Ist-order estimators:
{ Cupt 0
if Eq. (14) holds,
à
(15)
otherwise.
Here, dugi is the inverse-vares variance-optimal weight computed from the empirical variances of the Oth-order and sst-order gradients. In other words: If the test passes, we assume that (A1) and (A2) both hold and can therefore exploit the lower variance of the Ist-order estimator through IVW. If the test fails, we set a = O, reverting to a purely Oth-order estimator to avoid biased sst-rrder gradients.
Summary. DDCG utilizes the Ist-order gradient's lower variance wherever it is safe to do so. Our two assumptions (All) accurate empirical variance estimation and (A2) local smoothness-ensin that IVW is likely reliable. By checking Eq. (14), we detect plausible violations of either assumption. Failing this test triggers a fallback to safe Oth-order methods. In practice, this mechanism obviates the need for extensive hyperparameter tuning: aside from 5 (which controls confidence) and (which bounds how non-quadratic the function may be), the method remains largely automatic.
Comparison with AoBG. Our DDCG method and AoBG share the idea of constructing a statistical estimator for bias; however, a crucial difference is that AoBG uses the dlog)(7:0) terms in the ae
notoriously noisy REINFORCE estimator to construct a confidence interval. In contrast, our estimator in Eq. (14) uses only the function value and gradient variances. Consequently, in motivational toy tasks, the estimation of our bounds is d times more efficient than that of AoBG (Appendix D), where denotes the number of dimensions.
4.2 STEPWISE INVERSE VARIANCE WEIGHTING (IVW-H)
We adopt stepwise (per-step, per-action) inverse-variance weighting scheme. Let E (0,..,, 1} index time steps, n index actors (parallel rollouts), and let bold symbols denote action-dimennionall vectors in R.A, For each n), let Io.t.n and -1,r.n be the Oth- and Ist-order gradient vectors. We estimate empirical variances across actors at fixed t elementwise,
to.t.a = V.l
01,ca = Vnlli.mma)
(16)
IVW-H assigns a per-step. per-dimension IVW weight
60,ta
òi,a 60.t. + 01,t.
E 10, 11,
(17)
and forms the combined gradient elementwise as go,t,n.e Ôiaa I1 + (1 dv,,)) go.t.
(18)

---

Published as a conference paper at ICLR 2026
The combination is applied elementwise over (t, n, a) and then backpropagated through the policy network parameters. Following prior practice in total propagation estimators (Parmas, 2020), variance actoss actors at fixed (t, a) yields an efficient and stable estimate that aligns with per-step aggregation in trajectory optimization. This action-space formulation mirrors the Total Propagation X algorithm (Parmas al., 2023): first form composite gradient with respect to actions and then backpropagate it through the policy network. In principle, full TPX implementation should be the stronger estimator, and we expect it to outperform IVW-H when it can be implemented efficiently; however, TPX can be cumbersome to realize in practice due to simulator-specife implementation details. For reference, TPX is implemented in the Proppo framework (Parmas and Seno, 2022). Crucially, the batched actors at each time step provide the sample dimension needed for the empirical variances without introducing extra simulator calls, so the wall-clock cost of IVW-H comparable to that of pure Ist-order baseline. This stands in contrast to parameter-space IVW implementations such as the one reported in GIPPO (Son et al., 2023), which require additional simulator evaluations and were observed to be much slower and less effective. The pscudocode of the algorithm is provided in Appendix E.
5
EXPERIMENTS
5.1 OVERVIEW
We pursue two goals: (i) re-examine AoBG in explicit empirical-bias settings and evaluate DDCG in the same regimes; (ii) test whether variance-not bias-is the practical bottleneck on standard continuous-control benchmarks via IVW-H. Unless otherwise noted, all curves are averaged over multiple random seeds; the number of "trials" reported Appendix K coincides with the number of seeds for each setting.
Part Empirical-bias regimes (re-evaluating AoBG and validating DDCG). We revisit settings where empirical bias is known to arise and analyze AoBG's behavior (including the trajectory of the weighting parameter or) alongside IVW and baselines. We then evaluate whether DDCG improves outcomes under the same conditions. Following the original setup. compare five approaches: Oth-order grad, Ist-order grad, AoBG (parameter 7). IVW, and DDCG (parameter e with statistical test confidence 8 0.05). Unless otherwise noted, DDCG sensitivity is reported in Appendix H. The function-optiimizatinn (toy) experiments supporting the landscape analysis and a-selection diagnostics are in Appendix F.
unified hyperparameter 0.3;
Part II: Practical continuous control (IVW-H). To probe whether empirical bias is the primary issue in practical settings, we conduct experiments in differentiable physies with MuJoCo-style tasks (CartPole, Hopper, Ant), following prior usage in GIPPO and SHAC. In these domains, we hypothesize that explicit discontinuity detection is largely unnecessary because standard variance weighting (IVW) is already sufficient. Indeed, our sensitivity analysis (see Figure 16 in Appendix I) reveals that AoBG does not outperform simple IVW, implying that empirical bias is not the dominant bottleneck. Therefore, rather than applying DDCG, we focus on demonstrating that IVW-H- simple, computationally efficient stepwise update-is sufficient to outperform complex state-of-the- art baselines like GIPPO. We compare Ist-order grad, Oth-order grad, AoBG, IVW, IVW-H (our per-step, per-action IVW), and GIPPO.
5.2 DIFFERENTIABLE SIMULATION TASKS
First, we examine tasks that model physical systems with contact and friction. The setup was repli- cated using Suh (2021)'s code, implemented in Suh al. (2022), enabling direct comparison. Tasks fall into three categories: Landscape Analysis, Trajectory Optimization, and Policy Optimization. AoBG relies on tuned parameters; DDCG fixes 0.3. Since their paper lacks specifics, we set AoBG to the default parameters in the code. Refer to Appendix K for detailed parameter settings.
5.2.1 LANDSCAPE ANALYSIS
Experimental Setup. We study discontinuous landscapes to quantify estimation error and a selection, and we perform landscape optimization while visualizing cost convergence and a for AoBG, IVW, and DDCG (the visualization and IVW comparison were not included in Suh al. (2022)). We use
7

---

Published as a conference paper at ICLR 2026
0.80
0.59
0.21
Coot (log soale)
0.00
0.5
325
0.7
1.29
1:50
-0.02
0.00
0.000
00.09
-3200
-0.00-
0.015
4(Atr)
-0.900
-0.010
-0.12
0.00
0.25
2.50
0.76
1255
1.50
0.00
0.25
6.50
0.76
1,00
1.25
1.50
200
430
10.0
1.0
1.2
0.-
0.0
2.50
0.00
0.25
0.50
0.75
1.00
1.50
Angle throww (6)
Augle throww
Iteratione
Londacspe (1000 sumplos)
(b) with Landarapo (10 sacpppss)
Boll with Optimization (50 sumplos)
Mivende Grad
--- Ie-ender Grad
Auaii
DMW
JROM (Dves)
Figure Ball with Wall. Columns 2: top row shows the square root of estimation errors (scaled to match the previous study), middle row shows the cost function, and bottom shows selection. Column 3: optimization cost and a selection.
two tasks that exhibit ietisionniinuced discontinuities: Ball with Wall (maximize travel distance with impacts) and Momentum Transfer (maximize angular momentum transfer). For brevity we report Ball with Wall the main text and defer Momentum Transfer to Appendix G. Both tasks follow the setup of Suh et al. (2022) for fair comparison with AoBG.
Findings. For larger sample sizes (N 1000) Figure 2(a), IVW remains biased near collisions due an overconfident Ist-order component. Both AoBG (> 0.005) and DDCG detect these discontinuities and reduce the weighting parameter a. For smaller sample sizes (N 10), Figure 2(b) shows that AoBG's fixed becomes overly conservative, with dropping to zero, underutilizing available gradient information. In contrast, DDCG continues to detect discontinuities robustly using the same parameters. The cost convergence in Figure 2(c) confirms that both AoBG and DDCG avoid collisions by shifting toward the Oth-order estimator. Similar trends are observed the Momentum Transfer task. A detailed complete results for Momentum Transfer-is provided in Appendix G.
the variance and bias of the estimators, well as
5.2.2 TRAJECTORY OPTIMIZATION
In trajectory optimization, a sequence of control inputs is optimized for a known environment and initial conditions. We evaluated two tasks, Pushing and Friction, where contact and friction make 1st-order gradients inaccurate.
Pushing. Two rigid bodies collide with varying spring constants k: smaller k results in "soft" collisions, while a larger & leads to "stiff" ones. We apply force to the first body to minimize the second body's distance to the destination. AoBG was tuned per stiffness. For soft collisions, 1000 (the original parameter was extremely large and effectively loosened the constraint so that AoBG always used IVW, so we used smaller value). For stiff collisions, we set ? 10 For DDCG, = 0.3. Figure 3(a) and (b) show that under soft collisions, both AoBG and DDCG favor Ist-order gradients. In the low-sample setting (Figure 3(b),, AoBG conservatively relies on the Oth-order component, failing to leverage faster sst-order convergence, while DDCG continues using 1st-order gradients. Under stiff collisions, shown in Figure 3(c), we initially expected first-order gradients to be biased. However, we see that both AoBG and DDCG optimized by selecting values near IVW indicating that the stiffness caused variance instead.
Friction. Two overlapping objects interact under Coulomb friction, where static and dynamic friction cause abrupt transitions at near-zero relative velocity. A force applied to object to move object 2 toward the goal. In the original code, AoBG was not properly tuned, preventing effective use of 1st-order estimator; consequently, we re-tuned AoBG (y=30000). Furthermore, to enable clearer assumed larger sample size than in the code (N=1000. For DDCG, set c= 0.3. Figure 4(a) and (b) show that the Ist-order estimator and IVW stall once friction thresholds crossed. AoBG and DDCG detect and mitigate these discontinuities by shifting more weight
sample comparisons,
8

---

Published as a conference paper at ICLR 2026
7.8
Cost [Jog suale)
Coal [Dog strale)
Cost (log scsle)
7.8
7.8
7.4
7.4
7.2
7.2
7.0
9.0
6.8
7:0
200
400
200
400
800
200
300
1.0
1.00
0.5
0.5
0.50
0.00
200
Itrrations
Iterations
Iterations
(a) Pashiing Soft (100 samples)
(b) Pushing Soft (3 sumples)
(c) Pushing Statf (1000 samples)
orde Grad
-
1st-arder Grad
AoBG
IVW
DDCG (Ours)
Figure Pushing, Columns 2: soft collisions with different samples; Column stiff collisions.
6.5
Cost (log senle]
Cost. acsll)
Cost (log scsle)
5.0
94
4.5
9.2-
8.2
8.0
4.0
20
00
10
100
1.0
200
w/WW MMM M
0.75
AVD
0.5
0.0
0.00
10
20
30
40
10
20
40
50
00
100
150
200
Iterations
Iterations
Iterations
Friction (100 samples)
(b) (5 sumples)
Tennis (1000 samples)
Och-order Grad]
1st-order Grad
AoBG
IVW
DDCG (Ours)
Figure 4: Columns 1, 2: Friction with different samples; Column 3: Tennis.
to the Oth-order. When reducing the sample size, as in Figure 4(b), AoBG's performance degrades unless y is re-tuned, while DDCG maintains robustness against small-sample noise.
5.2.3 POLICY OPTIMIZATION
Tennis. Policy optimization adjusts the parameters 0 of state feetbefccbacck controller me. The policy gradient obeys VgJ = V JJ Jn, where J, = au/00 the policy Jacobian. Tennis, the agent steers paddle bounce incoming ball toward target. We optimize linear policy of dimension 21 over horizon = 200. Ball-paddle impacts create discontinuities, making the gradient unreliable in rough regions. Within DDCG (Sec. 4.1), each V is instantiated as VuJ, and the empirical variance é in Eq. (12) is computed over samples of V Vn.J We compare AoBG (> = 1000) and DDCG 0.3). Figure 4(c) shows that Ist-order and IVW stall, whereas AoBG and DDCG detect nonsmooth regimes, revert to Oth-order updates, and continue improving, AoBG and DDCG achieve identical final performance.
5.3 CONTINUOUS CONTROL BENCHMARKS
Experimental Setup. We evaluate Oth-order grad, 1st-order grad, AoBG, IVW, IVW-H, and GIPPO on MuJoCo-style tasks (CartPole, Hopper, Ant). Simulator and training hyperparameters follow GIPPO (Son et al., 2023). For AoBG, we set the hyperparameter y separately for each task based on preliminary sweeps: 7 = for CartPole, 7 10th for Ant, and r = 10s for Hopper. To probe whether empirical bias is the dominant issue under harder contacts, modify only the normal

---

Published as a conference paper at ICLR 2026
-1000
4000
-2000
3000
Re Reward
Reward
Reward
<000
2003
-1000-
-1000
0.25
0.50
0.75
0.4
1.2
Step
Step
Steep Hopper
Cartpolo
(b) Ant
(e)
godd
goal
(Our))
Figure 5: Episodic reward vs. environment steps on three MuJoCo-style tasks. Curves show the mean seeds; shaded bands indicate the empirical standard error.
contact stiffness (contact ke). Specifically, for Ant we set contact_ke = 4.0 x 10 (10x the GIPPO value), and for Hopper we set contact ke = 1.0 10 (50x). For completeness, we also report results under the original (unmodified) contact parameters in Appendix J, where both GIPPO and IVW optimize reliably and IVW-H matches or slightly improves upon IVW.
Experimental Results. CartPole (Figure 5(a)): the Oth-order baseline underperforms, while sst-order, AoBG, IVW, IVW-H, and GIPPO reach similar final rewards. Ant (Figure 5(b)): IVW-H attains the best returns; AoBG performs similarly to IVW (unable to outperform it even with tuning). IVW and GIPPO are comparable and clearly above Ist-order-only and Oth-order-only, which struggle. Hopper (Figure 5(c)): Oth-order surpasses Ist-order-only; GIPPO fails optimize; AoBG and IVW perform well, and IVW-H further improves upon IVW. Overall, these results indicate that variance control via stepwise IVW-H is often critical than explicit bias detection these benchmarks.
5.4 SUMMARY OF EXPERIMENTAL FINDINGS
Empirical-bias settings. In explicitly discontinuous regimes, IVW and the 1st-order estimator exhibit clear accuracy degradation near nonsmooth events. By contrast, AoBG and DDCG avoid failures by down-weighting ss--order information in such regions. However, inspecting AoBG's trajectories indicates that its behavior largely governed by heuristic parameter choices, with a wide operating range across tasks. In particular, AoBG requires task-specific 7 values that vary widely across our setups, with E [5 x 10-3, 10%]. DDCG maintains robustness under unified parameter and continues to function reliably even with small sample sizes; in fact, performance was essentially unchanged for any [0.1, 0.9].
Practical continuous control. In MuJoCo-style experiments with elevated contact, the IVW-H implementation achieves strong performance and consistently improves over standard IVW. Contrary the explicit empirical-bias settings above, these results suggest that variance, rather than empirical bias, is the dominant issue in these benchmarks; a practical per-step implementation such as IVW-H is sufficient to solve the problem effectively. On the MuJoCo-style tasks we consider, IVW-H matches or improves upon both IVW and the GIPPO composite-gradient baseline, without requiring any explicit empirical-be detection.
6 CONCLUSION AND DISCUSSION
This work primarily re-examines AoBG's claims under explicit empirical-bias regimes. Reproducing the original settings, we confirm that empirical bias indeed creates failure cases for variance-baver mixing, and we show that DDCG- while following the same protocol-achivver achieves more robust behavior with a unified hyperparameter by statistically detecting nonsmooth regions and switching estimators accordingly. As a practical complement, we introduce IVW-H, faithful per-step IVW implementation. On the MuJoCo-style benchmarks we study, IVW-H performs strongly without an explicit bias-detection scheme, indicating that in these environments variance control, rather than bias handling, is often the dominant practical concern, in contrast to the explicit empirical AoBG tasks where DDCG brings clear gains. Future work will broaden the task suite and deepen diagnostics to further delineate when bias- focused mechanisms are necessary beyond such practical implementations.
10

---

Published as a conference paper at ICLR 2026
AUTHOR CONTRIBUTIONS
Ku Onoda: Ran the DDCG experiments, as well as part of the experiments on the DFlex simulator tasks, discussed the interpretation of the results and wrote the paper.
Paavo Parmas: Conceptualized the idea; derived most of the equations, including the DDCG method; made the first IVW implementation in the DFlex simulator tasks; proposed the detailed implemen- tation of IVW-H; oversaw and supervised the project; discussed the interpretation of the results; comments and editing the writing.
Manato Yaguchi: Implemented IVW-H on the DFlex simulator tasks based on the existing IVW implementation and proposal of IVW-H by PP, and ran most of the DFlex simulator experiments. Some comments on the writing.
Yutaka Matsuo: PI of lab, funding acquisition, overall supervision in the lab.
ACKNOWLEDGMENTS
Paavo Parmas was supported by JST ACT-X, Japan, Grant Number JPMIAX23CO.
REFERENCES
Ronald J Williams. Simple statistical gralient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-55 256, 1992. 1,3
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterizzttin trick. Advances in neural information processing systems, 28, 2015, 1,3
Wonyeol Lee, Hangyeol Yu, and Hongseok Yang. Reparameterization gradient for non-differentiable models. In Advances in Neural Information Processing Systems, pages 5553 3553-556,,00 2018.
Paavo Parmas and Masashi Sugiyama. A unified view of likelihood ratio and reparameterizzatoo gradients. In International Conference on Artificial Intelligence and Statistics, pages 1078-4086.
PMLR, 2021. 1,2
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. PIPPS: Flexible model-based policy search robust to the curse of chaos. In International Conference on Machine Learning, pages 4062-4071, 2018. 2,
Paavo Parmas, Takuma Seno, and Yuma Aoki. Model-based reinforcement learning with scalable composite policy gradient estimators. In International Conference on Machine Learning, pages 27377. PMLR, 2023. 1, 3,3,222 4.2
Hyung Ju Sub, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning, pages 20668-20696. PMLR, 2022. 1, 2, 3, 3, 5.2, 5.2.1
Feng Gao, Liangzhi Shi, Shenao Zhang, Zhaoran Wang, and Yi Wu. Adaptive-gradien policy optimization: Enhancing policy learning in non-smooth differentiable simulations. In Forty-first International Conference on Machine Learning, 2024. 1, 2, A
Sanghyun Son, Laura Zheng, Ryan Sullivan, Yi-Ling Qiao, and Ming Lin. Gradient informed proximal policy optimization. Advances in Neural Information Processing Systems, 36:8788-888 8814, 2023. 1, 2, 4.2, 5.3, A
Andreas Griewank and Andrea Walther. Introduction automatic differentiation. In PAMM: Proceedings in Applied Mathematics and Mechanics, volume 2, pages 45-49, Wiley Online Library, 2003. 2
Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. In 202/ IEEE International Conference Robotics and Automation (ICRA), pages 9474-9481. IEEF, 2021. 2
11

---

Published as a conference paper at ICLR 2026
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a a differentiable physics engine for large scale rigid body simulation. arXiv preprint orX.v:2106.13281, 2021. 2, A
Justin Carpentier and Nicolas Mansard. Analytical derivatives of rigid body dynamies algorithms. In Robolics: Science and systems (RSS 2018), 2018. 2
Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact.
ACM Transactions on Graphics (TOG), 39(6):1-15, 2020. 2
Keenon Werling, Dalton Omens, Jeongseok Lee, loannis Exarchos, and C Karen Liu. Fast and feature-complete differentiable physics for articulated rigid bodies with contact. arXiv preprint arXiv:2/03.16021, 2021." 2
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojeiech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. In International Conference on Learning Representations, 2022.
Peter Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75-84, 1990. 3
Albert S Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics, 22(2):507-560, 2022. 3
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems, pages
2015. 3
Ming Xu, Matias Quiroz, Robert Kohn, and Scott A Sisson. Variance reduction properties of the reparameterization trick. In The 22nd international conference artificial intelligence and statistics, pages 27 1711-2720. PMLR, 2019. 4.1
Justin Domke. Provable gradient variance guarantees for black-box variational inference. Advances in Neural Information Processing Systems, 32, 2019. 4.1
Paavo Parmas. Total stochastic gradient algorithms and applications to model-based reinforcement learning. PhD thesis, Okinawa Institute of Science and Technology Graduate University, 2020. 4.2
Paavo Parmas and Takuma Seno. Proppo: message passing framework for customizable and composable learning algorithms. Advances in Neural Information Processing Systems, 35:29152- 29165, 2022. 4.2
H. J. Suh. Aobg official code repository:alpha_gradee gradient. GitHuu repository, https://gittut: com/ http//jsshh///ee hatpaalpaaaaagaeeeeea gradient, 2021. 5.2
Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. International Conference on Muchine Learning, pages 7805-7817. PMIR, 2021. A
Eliot Xing, Vernon Luk, and Jean Oh. Stabilizing reinforcement learning in differentiable multiphysics simulation. arXiv preprint arXiv:2412. 12089, 2024. A
Shenao Zhang, Wanxin Jin, and Zhaoran Wang. Adaptive barrier smoothing for first-order policy gradient with contact dynamics. In International Conference on Machine Learning, pages 41219- 41243. PMLR, 2023. A
Clemens Schwarke, Victor Klemm, Jesus Tordesillas, Jean-Pierre Sleiman, and Marco Hutter. Learn ing quadrupedal locomotion via differentiable simulation. arXiv preprint orXiv:2404.02888 2024.
A
12

---

Published as a conference paper at ICLR 2026
APPENDICES: DOES "DO DIFFERENTIABLE SIMULATORS GIVE BETTER
POLICY GRADIENTS?" GIVE BETTER POLICY GRADIENTS?
Extended Related Works
14
Infinite Variance Example
14
C Proofs
15
D Variance of the AoBG vs. DDCG Test Statistics
17
Pseudocode for IVW-H
19
F Function Optimization Tasks
20
G Additional Experiments
21
G.I Ball with Wall Task
21
G.2 Momentum Transfer Task
22
H Sensitivity analysis on the Parameter in DDCG
23
H.1 Ball with Wall Landscape
23
H.2 Sigmoid Function
................................
23
H.3 Optimization Problems
23
I Sensitivity analysis on the Parameter y in AoBG
26
J MuJoCo-style Tasks Results with Default Contacts
29
K Parameters
29
13

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORKS
Policy Optimization with Differentiable Simulation. In this appendix, we review additional research that leverages differentiable simulators for policy optimization and clarify the positioning of our work within this broader context.
Policy Optimization via Differentiable Simulators (PODS) (Mora al., 2021) refines policies using 1st- and 2nd-order updates derived from analytic gradients of the value function with respect to the policy actions. Analytic Policy Gradient (APG) (Freeman et al., 2021) directly computes policy gradients from simulator-provider analytic derivatives. These approaches do not explicitly consider discontinuities.
Several methods attempt to smooth the objective itself. Short-Horizon Actor-Critic (SHAC) (Xu al., 2022) truncates trajectories short horizon and uses a terminal value function smooth the objective while exploiting analytic gradients. Soft Analytic Policy Optimization (SAPO) (Xing al., 2024) adopts maximum-entroop RL framework and scales SHAC-style differentiable RL to deformable-bog body tasks, achieving superior performance over other methods on manipulation and locomotion benchmarks.
Other studies mitigate the effects of discontinuities by re-weighting analytic gradients rather than detecting them directly. Adaptive-Grade Gradienn Policy Optimization (AGPO) (Gao al., 2024) analyzes batch-gradien variance and switches to a surrogate Q-function, ensuring convergence and robust- ness under non-smooth MuJoCo-style dynamics. Gradient-Informee Proximal Policy Optimization (GIPPO) (Son et al., 2023) introduces an adaptively weighted a-policy to attenuate high-variance or biased analytic gradients, yielding consistent gains over PPO in function optimization, physics, and traffic control domains. While these methods alleviate discontinuity issues, they do not explicitly detect discontinuities.
A complementary line of work introduces explicit smoothing to handle non-smooth dynamics. Adaptive Barrier Smoothing (ABS) (Zhang et al., 2023) alleviates stiffness in complementarity- based contact models by adding barrier-smoothed objectives with an adaptive central-path parameter, jointly controlling gradient variance and bias for stable sst-order policy gradients. By smoothing contact interactions, amalytic-gradient methods such as SHAC have been applied successfully to learn physically plausible quadrupedal locomotion (Schwarke et al., 2024).
B INFINITE VARIANCE EXAMPLE
In this appendix, we provide a simplified example illustrating how a gradient estimator can exhibit infinite variance under small -robabbboyyyllityy event. Suppose we have a random gradient g(w) taking value 9I with probability p and 0 otherwise (with probability p). Let be the mean of this random gradient. Then,
G 91 p
E g) p
(19)
Next, compute the second moment:
G
G²
E p o (1- p.
(20)
p
The variance V[g] is given by:
G² G -² G² (1-1)- p
Viol = Bigg] -B()))
(21)
As p > 0, the term 1 goes to infinity, causing V[g] to blow up without bound. In practice, this p situation occurs when the estimator's nonzero gradients occur only in very small region of the parameter or state space, but those gradients can be extremely large. Although the unbiasedness condition P91 G still holds, the variance is unbounded when p approaches zero. This example closely parallels the situation where a Sigmoid gradients are near zero for most inputs (large (xx) and very large for small range (near x 0 with small temperature T).
14

---

Published as a conference paper at ICLR 2026
C
PROOFS
This appendix provides a step-by-step derivation of the key inequality Eq. (14) used in our proposed detection mechanism. We introduce a linear model for changes in gradient magnitude and then construct a quadratic model of f(x). These assumptions collectively yield a condition under which IVW is expected work well. If the condition fails, we revert the Oth-order gradient estimator to avoid potential bias or misleading variance estimates.
FIRST INEQUALITY:
Define linear model on the change in gradient magnitude between two points a and y:
L a| داا = [V) - V
(22)
such that the squared difference is minimized in expectation. We thus have:
i
E
[1||
ull [[= \ 0,
aT
(23)
Bi|lz 2ll [L|x yll || (z) vf(y)]l2)] O.
Define
A by
[V V ((v)ll - L|I v112-
(24)
Noting that
2 Vax E| ||| |
(25)
for arbitrary random variables, we can construct another equation involving the gradient differences and the above definition:
2 2 [f(z)]
[()))(()) - (
[L ull2 + A )²1
(26)
L [| v1111 + E E[
+ 2 E y y
0 from Eq. (23)
Using Eq. (25) again, and noting that E A > 0, obtain
2
(z)
> 2 E || 21
L f[v]
(27)
Furthermore, using V[ø] = Do², we get
(v)
La
(28)
Do2
where 2 is the variance of æ, and D) is the dimension.
SECOND INEQUALITY:
Using the same quantity L, we will construct another inequality by making a quadratic approximation of fx). Specifically, we define quadratic function with curvature L, given by
M(x:) E (f())) + v (æ: 11) I ||= -1112.
(29)
where vf E (=)) We also define Af = f(x) - .(()). Then, we have the equation
v[((x)] = v [Mxx + [=))]
v [+(x))+++///] (|)((())) + 2cov(Af(2)) n(x))
=
(30)
-v/v/' x
L |0 1112"
cov (v (x 1)) LL m1 +o3.
0
Covariance between odd and even.
15

---

Published as a conference paper at ICLR 2026
Now make another assumption o < cV [f())), where c [0, 1]. Then we have the inequality
V[f(a)
V
|| m1
+ +cV[f(x))
- M
1
T. 1ell
I. x 11 (1 -(V 4 m
V
-1 ml
2 - -1-1// [[(2)] A -
(31)
Gaussian distribution Eq. (34)
(2
1 4
2 [f([)()]]] V
2(1 c)V [f(x)] 2V V α
1
Dot
L > 2(1 (f()) 20²2
Vf
Do
Combining with Eq. (27), deduce:
|V We then replace f(z) with its empirical estimator e and incorporate the allowed estimation
v[V)(=)] 2 2 (1-c) 2
(32)
error e,:
i + Eu 2 (1 -1 (1 2
?
(33)
o²
which is the same Eq. (14) in the main text.
Note on | 1al2 and Gaussian assumption. Recall that
Vi1 14 E
14 [ (B[1 2=122))²
(34)
For Gaussian distribution, one derive explicitly that
|| 2 a and hence ||| - م [ = 30 o² 20 Note that Eq, (31), we used this particular result for Gaussian distributions. If different sampling distribution is used, would need to re-derive these statistical quantities or estimate them empirically from samples.
16

---

Published as a conference paper at ICLR 2026
D VARIANCE OF THE AOBG vs. DDCG TEST STATISTICS
Motivation. In the discontinuity detection test,
+DDCG uses the scaled empirical variance;
. AoBG forms confidence interval for the mean gradient via the score-function statistic.
The reliability of either test is controlled by the sampling variance of its statistic. We therefore compare their coefficients of variation
CoV(X) / [X]
Our goal to show
CoV = (dd) CoVDDOG,
meaning that the AoBG statistic is (cd) times noisier.
Toy set-up. Sample a d-dimensional vector x from the isotropic Gaussian (00, o2 and eval- Tij.
uate the linear reward f(x) Σj= TJ. AoBG measures bias via the score-function term V (ogp(=) while DDCG measures discontinuity via the (scaled) variance of f.
Score function term.
Because logp(z) --1x - p1l2 a212/(202) + const for a Gaussian,
logp(z))
1 1=
(35)
0p
0²
o2
Hence AoBG's per-sample statistic is
g(x) f(x) x σ²
[g) a²
kg) x
E [g] 1.
(36)
Step-by-step derivation of [/]. Let S E3 Tz. Then
52 E, x
E
[
11(())1
E [|9]|]]
(37)
04
Expanding yields
E
2
E
E [x]=]]
(cross terms with odd powers vanish).
(38)
For univariate standard normal 2, E [ = 3 and E [3]2]]] = ot when 21, 22 are independent. Hence
E [| d 4 + d(d 1)0*
d(d + 2).
(39)
04
Therefore
[gi
||||||
2|
|E [g] 2 d(d + 2) - d2 = =((d+ + 1) = O(d)).
(40)
DDCG statistic. Define
(1f((r)) o²
f
Because f(x) ~ N(0, do?),
[ = d
V [ = 2d
(41)
For batch of size the statistic used by DDCG is the sample mean
i 1 n
Σ
Zu.
(42)
Its sampling variance is therefore
V
7
2d
V
[0]
(43)
17

---

Published as a conference paper at ICLR 2026
Relative precision (coefficient of variation).
For any statistic X define
CoV(X) - VV]///XX
(44)
Hence
2d n
2
vd+I/vl
d
CDCDDGG
CoV
(45)
d
ndd
and their ratio scales as
CoV AoBG
Vd/n Odd).
(46)
CoVDDOG
d2/(nd)
v2
Thus AoBG's statistic is (d) times noisier than DDCG's, demonstrating DDCG's advantage in high-dimensional settings.
Monte-Carle confirmation. CoV quantifies the relative estimation error: it is the standard deviation of the statistic divided by its mean. We ran m 10000 independent batches of size n 1000 with
= 1; Table reports the empirical CoVs. The ratio CoVAA/// CDCDDC decays approximately as d, confirming the theoretical gap.
Table 1: Precision of the test statistics (n=1000, m=10000, r=1).
CoVDDCG
CoV AbBG
ratio
ratio
v2
4.49e-2
4.47e-2
1.00
1.41
16
1.1le-2
1.30e-1
11.7
16.6
64
5.56e-3
2.55e-1
45.5
64.4
128
3.90e-3
3.59e-1
92.2
130
18

---

Published as a conference paper at ICLR 2026
E PSEUDOCODE FOR IVW- W-
We implement practical composite update that combines Oth- and Ist-order policy gradients at the step and action-dimension level. The procedure summarized in Alg. 1.
Algorithm 1 IVW-H Policy Update (stepwise IVW)
Require: Horizon H, actors N, action dim. A; policy тo (Gaussian: μ4, σ); target critic V; ad- vantages A, via GAE; mask grad_start E {0, 10 for first-terms; optional pairwise noise/niitill-sttee sharing.
1: Define St, == grad_start((, n) and M := Ei= 11-1 En St, P number of trajectories (episode starts) in the batch
2: Rollout. For t = O,... H 1: compute (Me, (r) (s:),, sample (0, I), act a, tanh (1 + o O c2), step envs, cache (St, ac, M4, oz), and mark grad_staaaa startstttt at episode starts.
3: Advantages. Using {re. V}. compute GAE Ag; define the first-term sum over starts.
4: Losses exposing 91 and go-
RP/Ist-order loss: Cxp
Σ
Ar.a-
(normalize by number of trajectories M)
M
9t..
LR/Oth-order loss: (batch mean)
L1 t meant, Am, @ neglogPr..) with optional normalization of Á.
5: Parameter-leve gradients (per step and per dimension).
Backprop Lp Di,n,a, /00pp/00.....
Backprop Cu gin,a aLrr d
where ó E op and (t, n, a) index time, actor, and action dim.
6: Stepwise variance across actors. ei.. = Vn [g [grn,
02,00 = Vn [gn
7: IVW-H fusion (per step, per dimension).
átaa,
Gin.., = ar Din + ác,,,,) Dim
t 0 1,..
with ar, +00 wherever the DDCG gate suppresses gt.
8: Push to policy weights. Treat (G. as the target gradient on distribution parameters and perform a vacoornJacobian product through To to obtain VoL. Apply clipping needed and update O with Adam.
9: Critic. Fit V. by MSE to targets A, + (sz).
19

---

Published as a conference paper at ICLR 2026
F FUNCTION OPTIMIZATION TASKS
We measure the gradient estimation error on simple functions, revealing how each method adapts a under varying degrees of discontinuity and sample sizes.
Experimental Setup. We evaluate two functions (Sigmoid and Quadratic). For the sigmoid function, vary the temperature T. where smaller T yields near-discontinuities For both, also vary the sample size N to evaluate how each method performs with limited samples. For AoBG, the parameters y tuned separately for each function so that the methods perform well when the sample size is sufficiently large (around 100). Specifically, for the Sigmoid, we set 0.1, and for the Quadratic, we set y = 1.4. In contrast, DDCG uses the same settings (c 0.3) across all toy tasks. For detailed parameter settings, see Appendix K.
Findings. Figure 6(a), as the Sigmoid transitions become sharper (i.e., for smaller T), IVW starts to over-rely on Ist-order gradients and becomes biased. Both AoBG and DDCG detect these discontinuities and shift more weight to Oth-order, reducing error. However, as shown in Figure 6(b) and (c), AoBG tends to assign conservative weights the Oth-order component when sample sizes are small, causing the weighting parameter a to drop. This behavior arises from its sensitivity to the hyperparameter ",; without re-tuning, AoBG may underutilize useful st-orderr gradients, missing potential performance gains. DDCG, in contrast, maintains robust performance across both smooth and near-discontinuous regimes, achieving comparable better error reduction with fixed parameter setting.
10-1
Drrer
10-1
Eroee
10
00-3
10-4
10
10-4
0.8
0.5
0.50
0.50
0.25
0.25
0.00
10
10
10
10
10
Temperature
Sample Stee
Sumple Size
(1) Sigmoid (100 samples)
(6) Sigmoid (T
(e) Quadratic (d
- Otheerder grad
Ist-arder grdd
AniM3
IWW
+ DDDG (Ours)
Figure 6: Performance analysis for Sigmoid (Columns 1, 2) and Quadratic (Column 3) functions under varying temperatures and sample sizes. Top row: estimation errors (log scale) between true and estimated gradients for each method. Bottom row: weighting parameter a for each method, showing selection between Oth- and 1st-order gradients.
20

---

Published as a conference paper at ICLR 2026
G ADDITIONAL EXPERIMENTS
In this appendix, we provide more detailed results from the landscape analysis Section 5.2.1 1 for the Ball with Wall task, including variance and bias components of the gradient estimation error. We also present analogous results for the Momentum Transfer task, which could not be shown in the main text.
G.1 BALL WITH WALL TASK
As shown in Figure 7, near discontinuities, the Ist-order gradient estimator exhibits large bias that dominates the overall error. When the sample size sufficiently large (N = 1000), variance does not pose significant problem. However, with fewer samples, the Oth-order estimator tends to have higher variance, making it crucial to switch adaptively between 1st- and Oth-order estimates. DDCG achieves this by emphasizing the Ist-order gradient in smooth regions reduce variance while switching to Oth-order near discontinuities to avoid bias, thus maintaining low error across the entire landscape.
0.20
2.02
0.0
Cost [Dog Scale)
-0022
-0004
0.00
2.2
0.00
0.75
1.50
0.00
0.23
0.53
0.5
1.00
1.20
1.50
0089
Varuune
0.00-
Vaations
2.01
0.001
0.20
8.000
0.00
0.00
0.55
1.55
2.00
5.00
0:30
0.5
100
1.25
1.50
200
0.00-
2.00
1.20
0.01
1.0
3.000
0.000
Rlr.)
1.00
-022
Doll with Wall Optimization samples)
Anglee slroow (9)
Augle
(s) Dall wich Wall Landsrape (1000 sampless
(b) Ball woth Wall Landscape (10 samples)
Wllond gial
* Intandr gral
DVW
BEOS (Oum))
Figure 7: Ball with Wall task. Columns and 2: The first to third rows show the square root of estimation errors, variance, and bias, respectively (scaled to match the previous study). The fourth row shows the cost function, and the bottom row shows a selection. Column 3: Both the optimization cost and selection shown.
21

---

Published as a conference paper at ICLR 2026
G.2 MOMENTUM TRANSFER TASK
Figure shows similar results for the Momentum Transfer task. In terms of cost minimization, just in Ball with Wall, the Ist-order gradient and IVW struggle with discontinuities, whereas the other methods successfully circumvent them.
Viree
0.0
Ooot (lng
Varkunee
-0.8
-04
Ville
1.00
04
0.00
100
136
1.00
1.00
Rin,)
Rir.) B(Riv)
-02
-02
-025
0.00
025
0.51
075
1.0
125
135
0.00
025
0.50
140
122
1.50
100
2200
300
550
Trerartaco
(c) Momert Thansfer Optimization (50 samples)
5
0.5
0 05-
08
0.00
0.25
0.50
0.7
1.0
1.25
1.50
0.00
0.55
0.50
0.75
1.00
1.25
1.50
Angle thrown (9)
Angle thrown (9)
Mosment Transfer Lateleaaee sumples)
Thandhe Landhaae somples)
ged
goal
Figure 8: Momentum Transfer task. Columns I and 2: The first to third rows show the square root of estimation errors, variance, and bias, respectively (scaled to match the previous study). The fourth row shows the cost function, and the bottom row shows selection. Column 3: Both the optimization cost and a selection are shown.
22

---

Published as a conference paper at ICLR 2026
H SENSITIVITY ANALYSIS ON THE PARAMETER IN DDCG
In this section, we conduct sensitivity analysis on the parameter in our proposed DDCG method investigate how varying affects the detection discontinuities. We also clarify why 0.3 was chosen in this work.
H.1 BALL WITH WALL LANDSCAPE
Figure visualizes the Ball with Wall task landscape while varying e from to 1. Recall that c 1 means our test condition is always satisfied, so the method consistently applies IVW, disabling discontinuity detection. Conversely, falling back to the Oth-order estimator and leading more conservative updates. For any e + 1, the largest cost change near 0 0.7 is reliably detected. However, detecting milder discontinuity around 0 = 1.2 depends on c. Balancing these, we set e = 0.3 to avoid being overly conservative or too permissive, successfully detecting both major and moderate discontinuities.
0 imposes a strong smoothness assumption, frequently
0.002
0.004
0.006
0.008
R(T))
0,010
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.5
1.0
0.8
0.4-
0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.8
Angle thrown (0)
e= 0.1 c
0.5
0.7
Figure 9: Sensitivity analysis on c in the Ball with Wall task. The x-axis represents 0, and different values of e control the degree of discontinuity detection. Larger values of are less conservative, while smaller values lead to more frequent selection of Oth-order gradients.
H.2 SIGMOID FUNCTION
Figure 10 presents similar sensitivity analysis for the Sigmoid function, where we adjust its temperature parameter T. Smaller T values yield sharper transitions (stronger discontinuities). For 0, DDCG assumes stronger smoothness and thus tends remain conservative even the T 1 regime, resulting in larger estimation errors compared to larger values. On the other hand, when is close to 1, the method still detects strong discontinuities adequately, though it becomes less conservative in potentially nonsmooth areas.
H.3 OPTIMIZATION PROBLEMS
We report sensitivity sweep of the sole hyperparameter on the optimization problems: Pushing- Soft, Pushing-Stifr, Friction, and Tennis. Across these tasks, performance robust for a wide range E [0.1, the optimizer converges reliably and at similar rates. At the extremes, Rs 0 can be overly conservative on smoother tasks (e.g.. Pushing-Soop frequently falling back to Oth-order updates when first-order gradients are reliable, whereas sz 1 becomes very permissive and may under-detect mild nonsmoothness on strongly non-smooth tasks (c.g., Tennis, Friction). Hence, choosing c= 0.3 is representative rather than critical.
23

---

Published as a conference paper at ICLR 2026
OCborder gra
Oisonder godd
(tbondd goal
Tebortar and
13
Ievander aodd
Ialeeda goad
IVW
DOW
IV'
Erver
10-
IDDG (clom)
Emre
LIDENG (Uzzr)
DXIEII (Dkos)
10-
12"
10"
10
0.34
0.5
06-
DOCCT (Dkrs)
DENCG (clund
BDDG (Uan)
0.2
0.6
0.5
0.4
0.4
0.4
0.3
0.2
10-1
12
Temperature
Temperature
Temperature
(a) c c=0 0
(b) c c= 0.1
(c) c 0.3
0.bartke
Otloseka go
10'1
Errot
BDDG (Clr))
Ereur
DDDCS ((hrr)
10-1
DUIUT (Dons)
12
10-1
10
10
10
12.
Actili
Andiii
IVW
DUW
DOCG (Dazs)
DDCG (Chn))
DDOG (0an)
02
10'
10
12-2
12'
15
Tomparsture
Temprrature
Tengeratio
(d) = 0.5
(e) c = 0.7
(l) 0.9
Figure 10: Sensitivity analysis on e for gradient estimation error (log scale) and a selection in the Sigmoid function. The x-axis represents different values of the temperature parameter T, where smaller T indicates stronger discontinuities. Lower c values lead to conservative choices, while higher values make the method more permissive in discontinuity detection.
24

---

Published as a conference paper at ICLR 2026
Coet:log soale)
24-
c 00
Cont:
012
0.2
7.4
7.4
0.5
0.5
1.00
0.75
0.7
0.7
0.7
0.9
0.9
0.22
022
Olb-ankrr
Ollennkrr
leb-niie gnd
-.. Iab-niir god
[turat
Tourstionns
(a) Pushing-Sone
(b) Pushing-Stit Stiff
Cont(log amale)
0.8
Cost (log scale)
godd 1st ooder god
02-
6x110-
Oth-order grad 1st-order grad
012
+x10
20
20
130
250
200
= 0-4
1.02
c=0.1
0.75
C-0.3
06
t-0.9
5 0.50
077
C=1.0
0.25
0.00
e 89
0.02
50
100
260
Itorations
-0
30
Iteration
(c) Friction
(d) Tennis
Figure 11: Sensitivity of e on optimization tasks. Each panel shows optimization progress (c.g.. objective iterations episodes) for multiple values. Results indicate that non-extreme values yield near-identical performance; 0.3 convenient default rather than crucial choice.
Takeaway. For optimization problems considered, DDCG solves the tasks reliably for any non-extreme in [0.1, 0.9]. Thus, the method does not rely on finely tuned c using c= is safe and representative default.
Overall, we found 0.3 effectively balances performance in both highly discontinuous and smoothly varying scenarios; hence, we adopt it as the default setting.
25

---

Published as a conference paper at ICLR 2026
I SENSITIVITY ANALYSIS ON THE PARAMETER IN AOBG
We conduct the sensitivity analysis on the y parameter of the previous method AoBG. If I is large, AoBG will mainly use the IVW rule, y is small, AoBG mainly uses Oth-order estimates. Thus, in tasks where Oth-order estimates work well, 7 should be sufficiently small, and in tasks where 1st-order estimates are better, y has besufficiently large. Ball with Wall (1000 samples) requires roughly Figure 12 and Momentum Transfer (1000 samples) requires Figure 13, In the 3-sample Pushing Soft task, Oth-order methods perform poorly, and we find that y should be above roughly 50000 for good performance Figure 14. On the other hand, the Tennis task performs poorly when the gamma that large, requires roughly Figure As we can see, the optimal choice of yy varies widely between different tasks and also changes with the sample size.
0.02
VEnoo
0.01
0.00
1.2
1.5
0.75
0.50
0.25
0.00
0.2
0.4
0.6
0.8
1.2
1.8
Angle thrown (0)
Asilli 0.00
Aubii C.01)
AdBiI D.33)
ArBO
Figure 12: Sensitivity analysis on the parameter for AoBG in the Ball with Wall landscape analysis
(1000 samples). The figure shows the error for each input angle 0 and the corresponding selection.
7.5
Errrrr
5.0
0.0
0.2
1.2
1.00
0.50
0.25
0.2
1.2
Angle thrown (0)
AsBG (mmil)) originl)
AnBii (ym0.011
AsBG (ym10)
AvBG (=-0.01)
AvBCI (2-1)
Figure 13: Sensitivity analysis on the parameter 7 for AoBG in the Momentum Transfer landscape analysis (1000 samples). The figure shows the error for each input angle 0 and the corresponding selection.
26

---

Published as a conference paper at ICLR 2026
78-
1.2
1.0
Cost (log scale)
76
0.9
7.5
7.3
7.2
7.
200
400
203
Iverations
Iterations
AoBG (1-1000 original)
AoBG (y-5000)
AoBC (5-50000)
(+-3000)
(y-10000)
IVW
Figure 14: Sensitivity analysis on the parameter 7 for AoBG in the Pushing task with soft contact (3 samples). The figure shows the cost value evolution and the corresponding a selection across iterations.
5.5
1.00
Cost (log scale)
0.75
4.5
0.50
0.25
3.5
0.25
150
200
50
100
150
200
Iterations
Iterations
AoBG 10=10000 original) AoBG (y=50000)
AoBG (T=100000)
IVW
Figure 15: Sensitivity analysis on the parameter for AoBG in the Tennis task. The figure shows the cost value evolution and the corresponding selection across iterations.
27

---

Published as a conference paper at ICLR 2026
High-Contact Mu.ooo-sstyle Tasks. We conducted sensitivity analysis y for the Ant and Hopper tasks with high contact setting.
Figure 16 presents the learning curves and the evolution of across wide range of y values (Y E {10..... 10º) }). As shown in the results, while higher values of generally lead to better performance, AoBG does outperform the IVW baseline in either environment. If the performance limitation were primarily due to empirical bias, we would expect specific range of y to effectively mitigate this bias and surpass the baseline. However, the fact that IVW remains competitive or superior regardless of y tuning suggests that empirical bias is not the dominant factor hindering performance in these specific high-contase settings.
5000
1.0
3.00
1000
0.2
0.0
Step
X1I
Step
AnDO
AnDG 1000)
AnDG
AoDG
AnDG
(0000000)
1VW
AuRB (y= 100)
(a) Ant (High-contact scenario)
3500
3000
2500
Rewned
2000
1500
1000
0.2
500
0.0
0.50
0.75
3.00
1.50
0.25
0.75
3.00
1.25
3.50
Step
X107
Step
AndG (: 10)
Anddl ( - 100)
AdHG (-- 1000)
AndU ( 10000)
AcdtG ( 100000)
IVW
(b) Hopper (High--ontact scenario)
Figure 16: Sensitivity analysis on the parameter y for AoBG in high-contact environments. We compare AoBG with varying against the IVW baseline. The left plots show the learning curves (Reward), and the right plots show the evolution of the mixing coefficient a. Notably, even with extensive tuning of 7, AoBG does not consistently outperform IVW. This suggests that the performance bottleneck in these high-contact tasks likely not attributed to empirical bias.
28

---

Published as a conference paper at ICLR 2026
5000
6000
4000
Reward
Reward
4000
2000
2000
1000
0.4
0.8
1.2
1.6
Step
Step
(a) Ant
(b) Hopper
-
Oth-order grad
1st-ordes godd
IVW
AnBC
GIPPO
IW-II (Ours)
Figure 17: Episodic reward vs. environment steps on Ant and Hopper with default contact parameters.
Curves show the mean across seeds; shaded bands indicate the empirical standard error.
J MUJOCO-STYLE TASKS RESULTS WITH DEFAULT CONTACTS
Under the default MuJoCo-style contact settings (no change to contact_koe _ee,,,,, both GIPPO and IVW optimize reliably; IVW-H matches upon IVW across Ant and Hopper, while Oth-order gradients lag behind (see Figure 17).
K PARAMETERS
The following tables summarize the parameter settings used in our experiments. These parameters were chosen ensure consistency and reproducibility across all tasks.
Table 2: Sigmoid, Quadratic parameter settings
Sigmoid
Sigmoid
Quadratic
Parameter names
(Effect of Temperatures)
(Effect of Samples)
(Effect of Samples)
Common Parameters
Sample size
100
Standard deviation
Trials (Seeds)
500
500
500
AoBG
0.1
1.4
DDCG
0.3
0.3
0.3
Confidence level 5
0.05
0.05
0.05
29

---

Published as a conference paper at ICLR 2026
Table 3: Ball With Wall parameter settings
Landscape (1000 samples)
Landscape (10 samples)
Parameter names
Optimization
Common Parameters
Sample size N
1000
10
50
Standard deviation a
0.1
0.1
0.1
Trials (Seeds)
1000
1000
20
Iterations
1000
AoBG 7
0.005
0.005
0.014
DDCG
0.3
0.3
0.3
Confidence level
0.05
0.05
0.05
Table 4: Momentum Transfer parameter settings
Landscape (1000 samples)
Landscape
Parameter names
Optimization
(10 samples)
Common Parameters
Sample size N Standard deviation
1000
10
0.02
0.02
0.02
Trials (Seeds)
1000
1000
Iterations
5000
AoBG
0.2
0.2
0.2
DDCG
0.3
0.3
0.3
Confidence level 8
0.05
0.05
0.05
Table Pushing parameter settings
Soft Collisions (100 samples)
Soft Collisions (3 samples)
Parameter names
Stiff Collisions
Common Parameters
Sample size N
100
3
10
Standard deviation
0.1
0.1
0.05
Trials (Seeds)
100
100
20
Iterations
600
600
500
Spring constant A
10
10
1000
AoBG
1000
1000
10000000
DDCG
0.3
0.3
0.3
Confidence level 5
0.05
0.05
0.05
30

---

Published as a conference paper at ICLR 2026
Table Friction parameter settings
Trajectory (100 samples)
Trajectory (S samples)
Parameter names
Common Parameters
Sample size N
100
5
Standard deviation
0.1
0.1
Trials (Seeds)
15
Iterations
50
50
AoBG 7
30000
30000
DDCG
c
0.3 0.05
0.3
Confidence level 5
0.05
Table 7: Tennis parameter settings
Parameter names
Policy
Common Parameters
Sample size N
1000
Standard deviation a
0.01
Trials (Seeds)
4
Iterations
200
AoBG 7
1000
DDCG
0.3
Confidence level
0.05
31


Reviews:
[{"review_id": "F0r0InaNwT", "reviewer": "Reviewer_ace9", "summary": "This paper investigates a key challenge in reinforcement learning: how to best calculate policy gradients when using a differentiable simulator. It re-examines the trade-offs between two types of gradient estimators and proposes new methods to improve performance and robustness.", "questions": "I have no more questions beyond those mentioned in weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a significant and practical issue in reinforcement learning: how to effectively use differentiable simulators when their gradients are biased by non-smooth dynamics.\n- It provides a clear and constructive critique of the AoBG method. It compellingly demonstrates AoBG's key weakness: extreme sensitivity to its hyperparameter which requires extensive, task-specific tuning.\n- The paper proposes DDCG, a novel composite estimator. This method is a conceptual advance because its statistical test does not rely on the \"notoriously noisy\" 0th-order gradient estimator, which improves scalability and sample efficiency compared to AoBG.", "weaknesses": "- The paper's experimental validation is split. The proposed DDCG is tested on the \"empirical-bias\" tasks, while the simpler IVW-H is tested on MuJoCo.\n- The conclusion that variance, not bias, is the dominant issue in practice is based on only three MuJoCo tasks.\n- The DDCG method largely follows the \"test-and-fallback\" template of AoBG, with the main contribution being a replacement statistical test. This limits the paper's methodological novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dkB5HSUoxS", "reviewer": "Reviewer_Dv8Q", "summary": "The paper re-examines when analytic policy gradients from differentiable simulators truly help, proposing a lightweight statistical check to safely integrate analytic gradient with model-free gradient estimates via inverse-variance mixing", "questions": "- Equation 14 — The proposed check computes variance for the function values at the RHS. How to make sure that the quantity is accurately estimated in the first place? If it needs to be estimated from a sampled batch, then isn’t it running into the same issue and the argument circular?\n- Figure 4 (a-b): The estimated alpha values are quite different between AoBG and DDCG in these two cases, but why are the resulting costs almost the same? \n- Figure 4 (c): Why the 0th order gradient performs as well as DDCG and AoBG in this case? Does that suggest the evaluation is using too many samples such that the 0th-order gradient estimate is already accurate?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The proposed check is theoretically sound, clean, simple to verify, and provably more efficient than the existing AoBG approach; consequently, the estimates given by the proposed approach are provably less noisy than AoBG\n- Both adequate empirical results and theoretical justifications (except for one point I listed below in weaknesses)\n- I enjoyed the presentation, which walks the readers through the problem setup, existing challenges, and their solution smoothly\n- I like the experimental setups, especially that the authors design very focused experiments to test the proposed approach", "weaknesses": "- Line 147 “This estimator remains unbiased if R is continuous,” — I think this statement is misleading. The first order gradient is only unbiased if the dynamics model, reward function employed are perfect. It is just that in continuous regimes, the dynamics model normally tends to be more accurate. Similarly, “However, when R is discontinuous, the 1st-order estimator can be biased.” — the first-order gradient is almost always biased since there is barely a perfect model. \n- It seems that choosing a proper c is task-dependent. Also, assuming a near-quadratic model is restrictive \n- Why not also run AoBG in Section 5.3? I think it would be valuable to also see how AoBG performs in these continuous control tasks, especially given that performance gap between AoBG and DDCG is mostly small in previous tasks and that the paper is titled “Does “do differentiable simulators give better policy gradients?” give better policy gradients?”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XkcbhTAS9L", "reviewer": "Reviewer_MBBw", "summary": "This paper examines the limitations of differentiable physics engines when used to compute policy gradients in reinforcement learning.  In contrast with zero-order methods, such as REINFORCE and its descendants, differentiable physics engines allow direct first-order computation of the policy gradient.  This can reduce the variance of the gradient estimate, since it is less reliant on random sampling.  However, the bias of the gradient estimate may increase when the physical dynamics are not strictly differentiable everywhere - for example, at impulse events. Recent works employ a weighted average of zero- and first-order estimates to get the best of both worlds, but selecting the weights for the averaging is a non-trivial problem.\n\nThe present paper explores two methods to set those weights.  The first method (DDCG) uses statistical hypothesis testing to detect when the dynamics are near a discontinuity, in which case the zero-order estimate is used.  The second method (IVW-H) is a variant of a prior technique, \"inverse variance weighting,\" which computes the weights on a per-time-step basis.  The paper includes some theoretical considerations motivating these methods, as well as empirical comparisons with several prior works in several environments - including \"toy\" environments to aid the analysis as well as some Mujoco continuous control tasks.  On the one hand, the results show that explicit discontinuity detection with DDCG is competitive with prior methods, while being less sensitive to hyper-parameter tuning.  On the other hand, the Mujoco task results show that IVW-H, which does not explicitly detect discontinuities, performs comparably to or better than prior methods.  The paper concludes from the latter results that variance control may be more important than bias control, at least in some Mujoco RL benchmarks.", "questions": "- Can the authors explain more about Eq 14 - whether the true variance is/must be known for this method to work, why or why not?\n- Can the authors justify why they did not evaluate DDCG and IVW-H on the same benchmark?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- While I am not fully up to date on differentiable physics simulation, as far as I know the methods introduced in this paper are original.\n- The empirical results are rather comprehensive, involving multiple environments of varying complexity and comparison with multiple baselines.\n- The empirical results suggest that the new methods are more effective than past work, and likely to be a significant contribution in the sub-field of differentiable physics and first-order policy gradients.\n- Aside from certain issues described below, the paper is mostly clear and relatively easy to follow.", "weaknesses": "- I have a doubt concerning theoretical soundness.  The DDCG mixing weights are determined via Eq (14) which, according to the paper's notation, appears to use a true variance, not a sample estimate.  But then how is this equation used in practice, if the true variance is not known?\n- I also have a doubt about the experimental results and conclusions.  The paper makes a conclusion about the relative importance of variance vs bias control.  However, it does not seem there is any experiment that directly compares the two proposed methods on the same benchmark - DDCG for bias control, IVW-H for variance control.  So, I am not sure the empirical results support the overall conclusion of the paper.  I also wonder why there is no experiment evaluating both methods on the same benchmark.\n- In terms of presentation, much important information is pushed to the appendix.  I understand the page limit is tight, but it would be better if the main text were more self-contained.\n- Related to the previous two points, this paper appears to explore two orthogonal methods (DDCG, IVW-H), which dilutes the focus of the paper.  If each method were published in a separate paper, each paper would be more focused, and there would more room in the main text for a detailed analysis of each method.\n- There are some terms/symbols that are never defined or defined much later in the paper.  In particular:\n    - The acronym \"AoBG\" is never defined, I had to find it in one of the citations.  It should be defined at its first occurrence.\n    - In section 4.1, the function $f$ is mentioned abstractly without any specific example of what this function is.  A concrete example is given much later in 5.2.3, but the paper's clarity would benefit if one or more examples are mentioned when $f$ is first defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AJtbCKKayR", "reviewer": "Reviewer_ecYo", "summary": "The paper studies the problem of finding optimal trajectories using differentiable simulators. \n\nDifferentiable simulators allow us to differentiate dynamics (and rewards) w.r.t. actions, allowing efficient policy optimization. However, gradients are inaccurate in the presence of dynamic discontinuities (e.g., contact), posing a significant challenge for the use of gradient information in trajectory (or policy) optimization.\n\nPrevious work proposed mitigating the issue by combining zero-order (e.g., REINFORCE) and first-order methods (e.g., reparametrization gradients that differentiate the simulator). Zero-order gradients are unbiased, but typically suffer from high variance. First-order gradients, on the contrary, are subject to low variance, but can sporadically be subject to high variance in steep or non-differentiable regions. \n\nIn particular, the authors consider two algorithms: Inverse Variance Weighting (IVW) and Interpolation Protocol (AoBG).\n\nIVW uses the empirical variance of the estimator to reduce variance, allocating a higher weight to the estimator with the smaller variance. This method usually allocates a higher weight to the first-order method, but still fails to remove the bias. \n\nAoBG introduces an additional safeguard mechanism that aims to check the presence of bias by allocating more weight to the zeroth order in case the first order and the zeroth order estimates largely disagree. However, according to the authors, the hyperparameters introduced by AoBG are subject to high sensitivity and should be tuned for different environments/tasks. \n\nBuilding on these intuitions, the authors propose an algorithm that also builds on IVW and, similarly to AoBG, a \"filtering\" that allows one to revert to the zeroth-order information in the presence of bias. The authors notice that steep dynamics not only introduce a bias but might also harm the variance estimation of the estimator (i.e., the empirical variance might not be accurate). They devise a test that checks at once that the dynamics are not discontinuous, while the empirical variance is trustworthy (eq. 14). If that is the case, they rely on IVW weighting to perform the gradient estimation; otherwise, they revert entirely to relying on the zeroth-order estimator.\n\nThe authors show that a) their algorithm is less sensitive to estimation errors due to discontinuities (Figures 2 and 3); b) Figure 4 clearly highlights the ability of the developed algorithm to switch to the zeroth order derivative when needed (unlike IVW), while AoBG can be either too conservative or too lenient. Although the error difference in the ablation studies does not seem so significant, the policy optimization experiment (Figure 5) really shows the advantage of relying on good gradient estimation. This advantage is greater in high-dimensional, contact-rich tasks like Ant and Hopper, whereas it is negligible in the smooth, low-dimensional dynamics of the pendulum.", "questions": "Questions\n--------------\n\nI am unsure why the proposed method checks only the empirical variance of the first-order estimate. Typically, zeroth-order estimators also suffer from high variance, which can make the empirical variance less precise. Could the authors clarify that?\n\nSuggestions\n----------------\n\nA technical note: the baseline subtraction can make REINFORCE biased if it is estimated with the same samples. Look at this example, where b is the average reward (typical baseline subtraction):\n\n$$\\hat{g}^{(n)} = n^{-1}\\sum_{i=1}^n \\nabla \\log p(\\tau_i) \\left(r(\\tau_i) - \\hat{b}^{(n)}\\right)$$ with \n$$\\hat{b}^{(n)} = n^{-1} \\sum_{i=1}^n r(\\tau_i) $$\n\nFor $n=1$, $\\hat{g}^{(n)} =0$, clearly show that, unless the ground truth is always zero, this gradient estimator is biased. Of course, this issue is avoided when one uses two independent sets of samples to estimate the gradient and the baseline.\n\nI recommend adding this useful information to the text and explicitly writing how you compute the baseline subtraction for the 0th order.\n\nSection 4.2 could have a more meaningful title.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper analyses an important and relevant problem in model-based policy optimization arising from inaccuracies in differentiable simulators.\n\nThe authors present the state of the art well, explain the problems, introduce their solution, and build upon it. The reading is clear and interesting.\n\nThe presented method is sound. \n\nThe empirical analysis is very well done. The authors present three studies: 1) they check how a discontinuity affects the gradient estimators; 2) they check both pure trajectory optimization and policy optimization, showing gradient estimation errors, the weights of the estimators, and the total cost; and 3) the learning curve in a pure policy optimization scenario in three MuJoCo tasks. A hyperparameter sensitivity analysis is provided in the Appendix. \n\nThe results agree with the intuition built into the method.", "weaknesses": "This paper has a few weaknesses:\n\n1) Equation 14, which is the core of the method, is not really explained in the main paper. The authors should have devoted some effort in building an intuition for it, rather than relegating the proof to the appendix. I am aware that, in estimating the gradient of a function, the concavity (or convexity) has an impact on its variance, thus I think that building an intuition on eq. 14 should not be too hard.\n\n2) The acronym AoBG has never been introduced. \n\n3) Equations 4 and 6 seem to be one-sample estimates, but later in the paper, it becomes clear that they are utilizing n-samples. \n\n4) The notation in equation 3--6 is a bit too compact, hiding details and dependencies between variables. \n\n5) the number of seeds used is not explicitly stated in the main text (unless I have missed it). The tales in Appendix K report number of trials (which is typically high). I am unsure whether the trial correspond to the number of seeds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)