A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation
Authors: Xinjie Liu
Cyrus Neary
Kushagra Gupta
Wesley A. Suttle
Christian Ellis
ufuk topcu
David Fridovich-Keil
Venue: Accepted by TMLR ()

Abstract: Many reinforcement learning (RL) algorithms are impractical for deployment in operational systems or for training with computationally expensive high-fidelity simulations, as they require large amounts of data. Meanwhile, low-fidelity simulators—such as reduced-order models, heuristic reward functions, or generative world models—can cheaply provide useful data for RL training, even if they are too coarse for direct sim-to-real transfer. We propose multi-fidelity policy gradients (MFPGs), an RL framework that mixes a small amount of data from the target environment with a control variate formed from a large volume of low-fidelity simulation data to construct an unbiased, variance-reduced estimator for on-policy policy gradients. We instantiate the framework by developing a practical, multi-fidelity variant of the classical REINFORCE algorithm. We show that under standard assumptions, the MFPG estimator guarantees asymptotic convergence of multi-fidelity REINFORCE to locally optimal policies in the target environment, and achieves faster finite-sample convergence rates compared to training with high-fidelity data alone. We evaluate the MFPG algorithm across a suite of simulated robotics benchmark tasks in scenarios with limited high-fidelity data but abundant off-dynamics, low-fidelity data. In our baseline comparisons, for scenarios where low-fidelity data are neutral or beneficial and dynamics gaps are mild to moderate, MFPG is, among the evaluated off-dynamics RL and low-fidelity-only approaches,the only method that consistently achieves statistically significant improvements in mean performance over a baseline trained solely on high-fidelity data. When low-fidelity data become harmful, MFPG exhibits the strongest robustness against performance degradation among the evaluated methods, whereas strong off-dynamics RL methods tend to exploit low-fidelity data aggressively and fail substantially more severely. An additional experiment in which the high- and low-fidelity environments are assigned anti-correlated rewards shows that MFPG can remain effective even when the low-fidelity environment exhibits reward misspecification. Thus, MFPG not only offers a reliable and robust paradigm for exploiting low-fidelity data, e.g., to enable efficient sim-to-real transfer, but also provides a principled approach to managing the trade-off between policy performance and data collection costs.

Published in Transactions on Machine Learning Research (02/2026)
A Multi-Fidelity Control Variate Approach for Policy Gradi- ent Estimation
Xinjie Liut
zinjie-lnu@ueens.edd
The University of Texas at Austin, Austin, TX, USA
Cyrus Nearyl
cyrus. egrus.ncarySuub.crr
The University of British Columbia, Vancouver, BC, Canada
Kushagra Gupta
Saatarrrgguutzass.. edu
The University of Tezas at Austin, Austin, TX, USA
Wesley A Suttle
wesley, wtw.ee.ttrrarrr.....ommmmmmm mill
DEVCOM Army Research Laboratory, Adelphi, MD, USA
Christian Ellis
christian, www.riinnellssssstii.... www.::/aawutiit..tssss.. edu
The University of Texas at Austin, Austin, TX, USA DEVCOM Army Research Laboratory, Adelphi, MD, USA
Ufuk Topcu
utopcushuteras.co edu
The University of Teras at Austin, Austin, TX, USA
David Kridovich-Keil
dfte@utezas.co edu
The University of Teras at Austin, Austin, TX, USA
// Reviewed on OpenReview: https: // openreview. net/ forum? id= zλoOL
Abstract
Many reinforcement learning (RL) algorithms are impractical for deployment in operational systems or for training with computationally expensive high-fidelity simulations, as they require large amouns of data. Mcanwhile, low-fidelity simulators such as reduced-order models, heuristic reward functions, or generative world models can cheaply provide useful data for RL training, even if they are too coarse for direct sim-to-real transfer. We propose multi-ffdelit policy gradients (MFPGs),
RL [ramework that mixes a small amount of data from the target environment with control variate formed from large volume of low- fidelity simulation data to construct an unbiased, varianco-reduced estimator for on-policy policy gradients. We instantiate the framework by developing practical, multi-fidelts vari- ant of the classical REINFORCE algorithm. We show that under standard assumptions, the MFPG estimator guarantees asymptotic convergence of fudtii-iieeity REINFORCE to locally optimal policies in the target environment, and achieves faster saite- conver- gence rates compared to training with high-fidelity data alone. We evaluate the MFPG algorithm across a suite of simulated robotics benchmark tasks in scenarios with limited high-fidelity data but abundant off-dynamics, low-fidelity data. In our baseline comparisons, for scenarios where low-fidelity data are neutral or beneficial and dynamics gaps are mild to moderate, MFPG is, among the evaluated off-dynamics RT. and low-fidelity-only approaches, the only method that consistently achieves statistically significant improvements in mean performance over a baseline trained solely on high-fidelity data. When low-fidelity data become harmful, MFPG exhibits the strongest robustness against performance degradation
Equal contribution.
Project website: https://xiifieeliit www.iiiiiieiiigggggh.b...ebebbb wtt.ggittbbi../gg.ttit...

---

Published in Transactions on Machine Learning Research (02/2026)
among the evaluated methods, whereas strong off-dynamies RL methods tend to exploit low-fidelity data aggressively and fail substantially more severely. An additional experiment in which the high- and low-fidelity environments are assigned anti-correlated rewards shows that MFPG can remain effective even when the low-fidelity environment exhibits reward misspecification. Thus, MFPG not only offers a reliable and robust paradigm for exploiting low-fidelity data, e-g., enable efficient sim-to-real transfer, but also provides principled approach to managing the trade-off between policy performance and data collection costs.
1 Introduction
Reinforcement learning (RL) algorithms offer significant capabilities in systems that work with unknown, difficult-t--specffy dynamics and objectives. The flexibility and performance of RL algorithms have led to their adoption in applications as diverse controlling plasma configurations in nuclear fusion reactors (De- grave et al. 2022 piloting high-speed aerial vehicles (Kaulmann et al. 2023), training reasoning capabilities in large language models (Shao et al. 2024), and searching large design spaces for automated discovery of new molecules (Bengio et al. 2021 Ghugare et al. 2023). However, in many such applications, datasets must be gathered from operational systems or from high-fidelity simulations. This requirement acts as a significant barrier to the development and deployment of RT policies: excessive interactions with operational systems are often either infeasible unsafe, and generating simulated datasets for RL can be prohibitively expensive unless the sinulations are both cheap to run and carefully designed to minimize the sim-to-real gap.
On the other hand, low-fidelity simulation tools capable of cheaply generating large volumes of data are often available. For example, reducod-ordor models, lincarized dynamics, heuristic reward functions, and generalive world models all outpul useful information for RL, even when approximations of the target dynamics and rewards are very coarse.
Towards enabling the training and deployment of RL policies when expensive, high-fidelity samples are scarce, we develop novel mulli-fidelily RL algorithm. The proposed algorithm mixes data from the target enviromment with data generated by lower-fidelity simulations to improve the sample efficiency of high-fidelity data. The proposed framework not only offers novel paradigın for sim-to-real transfer, but also provides a principled approach to managing the trade-off between policy performance and data collection costs.
More specifically, we present multi-fideiity policy gradients (MPPGs), an RL framework that mixes small amount of data from the target environment with control variate formed from large volume of low- fidelity data to construct unbiased, variance-reduced estimator for on en-policy policy gradients. We further propose practical algorithm that instantiates the MFPG estimator with the classical on-policy algorithm
REINFORCE Williams 1992
MFPG Sumpling Mechanism
MFPG Policy Gradient Estimator
High-Fidelity Env.
Vo +( x
Current policy its
NW
ch log looaplaa
N comobined uajectey samples from eass.
Σ. of log mogaaill
N! 2o Nrh
Low-Fidelity Env.
Low- Low-Fidele Fidelity Policy Gradient Estimator
NI Σr, G¹ log (a
N low-idelity cajocticy samples
policy
Apply Policy Gradient Step
Figure 1: The proposed multi-fidelity policy gradient (MFPG) framework. At each policy update step, MFPG combines small amount of data from the target (high-fidelity) environment with large volume of low-fidelity simulation data, thereby forming unbiased, reduced-varies variance estimator for the policy gradient.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 1 illustrates the proposed approach. At each policy update step, the algorithm begins by sampling relatively small number of trajectories from the target environment, which may correspond to real-world hardware or to a high-fidelity simulation. We then propose a method for sampling trajectories from the low-fidelity environments such that the resulting action likelihoods are highly correlated with those from the high-fidelity trajectories. This method hinges on a nuanced approach to action sampling and is critical to the success of our approach. Next, the algorithm uses the low-fidelity environments to sample a much larger quantity of uncorrelated trajectories, which it uses alongside the previously sampled trajectories to compute unbiased estimate of the policy gradient to be applied. So long the random variables corresponding to the policy gradient updates between the high and low-fidelity environments are correlated, the approach is guaranteed to reduce the variance of the policy gradient estimates.
Theoretically, we analyze the proposed MFPG estimator for the REINFORCE algorithm and prove that, under standard assumptions, the MFPG estimator guarantees asymptotic convergence to a locally optimal policy (satisfying first-order optimality conditions) in the taryet (high-fidelity) cnvironment. Morcover, when high- and low-fidelity samples exhibit. nonzero correlation, MFPG achieves a faster finite-sample convergence rate than using high-fidelity samples alone.
Empirically, we evaluate the proposed algorithm on a suite of benchmark robotic control tasks where the high- and low-fidelity environments differ in transition dynamics, and additionally on a task where the low-fidelity reward is the negative (anti-coreelate)) version of the high-fidelity reward. We use 20 random seeds per setting and non- non-parametric bootstrap confidence intervals to assess statistical significance. In high-fidelity sample-scarce regimes, the results reveal three key insights: i) When the dynamics gap is mild, MFPG can substantially reduce the variance of policy gradient estimates compared to standard variance reduction via state-value baseline subtraction. ii) MFPG provides a reliable and robust way to exploit low-fidelity data. In our baseline comparison experiments, for scenarios where low-fidelity data are neutral or beneficial and dynamics gaps are mild to moderate, MFPG is, among the evaluated off-dynamics RL and low-fidelity-on approachos, the only mothod that consistently and statistically significantly improves the mcan performance over a baseline trained solely on high-fidelity data. When low-fidelity data become harmful, MFPG exhibits the strongest robustness against performance degradation among the evaluated off-dynamics RL and low- fidelity-only methods. Strong off-dynamics RL methods tend to exploit low-fidelity data more aggressively, and can achieve high performance at times, but can also fail substantially more severely. iii) MFPG can remain effective even in the presence of reward misspecification in the low-fidelity samples.
2 Related Work
This section provides concise overview of three lines of related work: (i) variance reduction in RL via control variates; (ii) multi-fidelity RL and fine-tuning using high-fidelity data for sim-to-real transfer; and (iii) off-dynamics RL.
Variance Reduction in RL via control variates. The method of control variates (CVs) is commonly
used for variance reduction in Monte Carlo estimation (Owen) 2013). In RT, particularly in policy gradient
(PG) methods (Williams 1992). there is a long history of using CVs to reduce variance and accelerating
learning, e.g., subtracting constant reward baselines [Sutton] 1984 Williams
1988 Dayan 1991 Williams)
1992 or state value functions (Weaver & Tao, 2001 Greensmith et al.) 2004 Peters & Schaal 2006 Zhao
ct al. 2011) from Monte Carlo returns. Even though state values are known to not be optimal for variance
reduction [Greensmith et al., 2004), they are also central to modern algorithms, e.g., actor-critic meth- ods (Schulman et al. 2017). More recently, additional CVs baselines have been studied, such vector-fone
CVs [Zhong et al. 2021), multi-step variance reduction along trajectories (Pankov| 2018 Cheng et al. 2020
and their more general form (Huang & Jiang] (020), and state-action dependent basclines (Gu et al. Liu el. al. 2018 Grathwohl el al. 2018 Wu el al.) 2018), which offer better variance reduction under certain conditions (Tucker et al., 2018). The literature on using CVs for policy gradients almost exclusively focuses on single-environment settings. In this work, we consider the problem of fusing data from multi-fidelity environments to construct low-variance estimators, cf. Section 4 Moreover, the proposed multi-fidelity approach can also be readily combined with existing single-fidelity CVs for further improved performance.
2017
3

---

Published in Transactions on Machine Learning Research (02/2026)
To the best of our knowledge, the only existing work leveraging multi-fidelity CVs for RL is Khairy & Balaprakash p024p. Our approach differs in two main ways. First, we propose a multi-fidelity PG algorithm for Markov decision processes (MDPs) with either continuous or discrete state and action spaces, whereas Khairy& & Balaprakash 2024) estimate state-action values in tabular MDPs without function approximation.
Second, we propose a novel policy reparameterization ion trick that enables the sampling of correlated trajectories across multi-fidclity environments. Crucially, this sampling schome improves the variance reduction of the proposed MFPG algorithm, and it directly supports continuous settings without restricting the MDPs. By contrast, Khairy & Balaprakashi (2024) require action sequences to be matched across multi-fidelityy environments, which requires additional assumptions about the MDP structure and transition dynamics.
Multi-fidelity RL and fine-tuning using high-fidelity data. Multi-fidelity methods for optimization, uncertainty propagation, and inference are well-studied in computational science and engineering, where it is often the case that multiple models for a problem of interest are available (Pcherstorfer et al. 2018). Such methods provide principled statistical techniques to manage the computational costs of Monte Carlo simula- tions, without introducing unwanted biases. However, the adoption of such techniques by RL algorithms is significantly more limited. Two bodics of literature which are closcly related to this work are multi-fidelityy RL (Cutler et al., 2015) and fine-tuning RL policies using limited target-domain data, e.g., in sim-to-real transfer (Smith et al. 2022). Multi-fidelity RL aims to design training curricula, i.e. decide when to train in each simulation fidelities, in order to train highly-performant policies for the highest-fidelity environment with minimum simulation costs. Typically, training begins in the lowest-fidelity simulators, with proposed mechanisms to decide when to transition to other fidelities based on estimated uncertainly (Suryan el al.
2023 in predicted 2017 Cutler et al.] 2015 Agrawal & McComb, 2024 Ryou et al.) 2024 Bhola et al.)
actions, state-action values, dynamics, rewards, and/or information gain (Marco et al. 2017). Similarly,
in sim-to-real and transfer learning settings (Taylor & Stone 2009 Zhao et al.) 2020: Tang et al., 2024
Da et al., 2025 Niu et al., (2024), the objective is to train a highly-performant policy for the real world using simulation data, sometimes supplemented with small amount of real-world data. Many approaches leverage models or value functions trained in simulation (source domain) to bootstrap fine-tuning in the real
world (target domain) Rusu et al. 2017 Arndt et al. 2020 Taylor et al.) 2007 Smith et al 2022]. guide real-world exploration YYin et al. 2025), or bidircctionally align state distributions between simulation and real-world agents to improve sample efficiency. Real-world data can also be used to refine the simulator
itself (Abbeel et al. 2006 Chebotar et al. 2019 Ramos et al. 2019p. The works above introduce new training paradigms and exploration strategics with data from different domains. In contrast, our approach proposes a new policy gradient estimator that incorporates cross-domain data. The proposed approach can be seamlessly integrated into existing paradigms that involve fine-tuning within target domains.
Off-dynamics RL. A particularly important setting is when multi-fidelity environments exhibit mis- matched transition dynamics. Classical approaches such as system identification (L.jung 1998) and domain randomization (Tobin al., (017)) address this problem from a dynamics modeling perspective, but typically roquire either access to high- fidcliiciitty dynamics model or a pre-specified distribution over dynamics param- eters. Other works (Desai et al. 2020b ab instead learn action transformations to compensate for transitions produced by low-fidelity dynamics.
Another line of work tackles the off-dynamics RL problem from a policy adaptation perspective, i.e., modi- fying policy learning with samples from multiple domains. DARC (Eysenbach et al.) estimates importance weights for low-fidelity transitions- vvi pair of learned classifiers modeling the likelihood ratio under high- and low-fidelity dynamics and uses these weights to augment the low-fidelity rewards. Guo et al. (2024) extend this idea with generative adversarial imitation learning. Similar in spirit to DARC (Eysenbach) et al.),, more recent works estimate the dynamics mismatch by comparing observed low-fidelity transitions with a corresponding high-fidelity prediction-cither through explicit target dynamics models to evaluate state-value consistency {Xu et al. (202)) or through latent representation learning (Lyu el. al.] (024a)). The estimated dynamics gap is then used to filter out unrealistic low-fidelity transitions (Xu et al. 2023) or to
augment the low-fidelity rewards by penalizing less likely transitions (Lyu et al., 2024a). Van et al. (2024) introduces a mechanism that re-weights and extrapolates low-fidelity transitions while augmenting rewards, in order to mitigate deficient low-fidelity data support under large dynamics gaps. A related line of work
4

---

Published in Transactions on Machine Learning Research (02/2026)
extends these ideas to offline RL, leveraging either online low-fidelity simulation data Niu et al. 2022
2025 2025
or large offline low-fidelity dataset (Liu et al. 2022 Lyu et al. 2025: Guo et al. 2025 Anonymous to complement small high-fidelity dataset.
In this work, we focus on the online, on-policy setting, where the agent has limited access to interact with the high-fidelity (target) environment. but can collect abundant low-fidelity samples for each policy update. We benchmark our proposed MFPG algorithm against DARC (Eysenbach et al.) and PAR (Lyu et al. 202-4a). A key distinction between MFPG and most existing off-dynamics RL methods lies in their fundamental design philosophy. Prior approaches often rely on regularizing low-fidelity samples for policy learning (Eysenbach) ct al.) Xu ct al.) 2023 Lyu et al.] 2024ap. In contrast, MFPG explicitly grounds policy learning in high- fidelity samples, leveraging low-fidelity data solely as a variance-reduction mechanism. This difference has important theoretical implications. Existing methods (Eysenbach et al. Xu et al. 2023 Lyu et al. 2024a)
bound the global suboptimality of policies deployed in the high-fidelity (target) environment; the bounds can degrade as the dynamics gap grows- example, when discrepancics in value functions (Xu et al. 2023) learned representations (Lyu et al.] 2024a) become more pronounced. DARC (Eysenbach al.) further assumes that policies optimal in the target domain remain near-optimal in the source domain. By contrast, MFPG requires no such assumption: low-fidelity samples are used exclusively for variance reduction- our asymptotic guarantees ensure that the learned policy is locally optimal (up to first order) in the high- fidelity enviromment, regardless of the dynamics mismatch. This design principle enables us to establish faster convergence rates for MFPG with REINFORCE when the correlation between low- and high-fidelity samples is non-zero. Empirically, MFPG demonstrates greater robustness than DARC and PAR. Morcover, unlike prior methods designed specifically for dynamics mismatch, MFPG can also extend to settings with reward misspecification, as illustrated in Section 6.4
3 Preliminaries
Our objective is to develop RL algorithms capable of leveraging data generated by multiple environments in order to efficiently learn a policy that achieves high performance in a target environment of interest.
Modeling multi-fidelity environments. We consider a high-fidelity environment and a low-fideiiyy en- vironment, each modeled by a finite-horizon MDP. In particular, the high-fidelity environment an MDP M (S, A, 7.F", Rh, TT), which we assume represents either an accurate simulator of the target envi- ronment, or the target environment itself. Here, S is the set of environment states, A is its set of actions, As, is initial distribution over states, 7 E [0, 1] is a discount factor, (''''ss, (, is the probability of transition- ing to state s' from state s under action a, r ~ Rh,s, a, s') defines the probability of observing a particular reward under a given state-action-state triplet, and T denotes the finite time horizon length. Similarly, the low-fidelity environment is an MDP M = (5, A, Asp R²', T), whose transition dynamics p' and reward function Re differ from those of M², and do not necessarily accurately represent the larget environment.
Assumptions on the multi-fidelity environments. We assume that the cost of generating sample trajectories in Mh is siguificantly higher than that of generating trajectories in M', whose cost we treat as negligible. This may be the case if Mh were to represent real-world hardware while M were to represent cheap simulations. In addition, the proposed MFPG approach requires that the low-fidelity simulator can be reset to user-specified states on demand, so that we can construct low-fidelity trajectories whose initial states match those of the high-fidelity rollouts and thereby obtaïn correlated trajectories.
Our objective is learn a stochastic policy Ta(a))), parameterized by E Rd, that achieves a high expected total reward in the high-fidelity environment M². Fixing policy To defines distribution over trajectories in both M and M' We denote trajectories in M by Th == si, aB ró. st, af r ..... s4. where sB A.y: af ~ ))))) shan ~ (-1s,, af), and r ~ (sh ab 14.1). Similarly, we use r' to denote trajectories sampled in M
Policy gradicnts. PG algorithms aim to maximize the performance measure J(O) == Er- [[()] the expected total reward along trajectories sampled from environment M under policy тe. They do by using stochastic estimates of the policy gradient VoJ(O) to perform gradient ascent directly the
5

---

Published in Transactions on Machine Learning Research (02/2026)
policy parameters (Sutton & Barto 2018). For example, the REINFORCE algorithm (Williams 1988
1992 uses Monte Carlo estimates of (xn-M(ss)[VoXX)) to estimate the policy gradient, where X
T Ei= Golog (a(a))))) is a random variable defining the contribution of each trajectory т to the overall policy gradient. Here, at; St, and G - ""="r= denote the selected action, the state, and the reward- lo-go al time & in trajectory respectively.
Different policy gradient algorithms use different expressions for X when estimating the policy gradient (o.g., G2 may be replaced with an advantage estimate, or X may be entirely replaced by a surrogate per- trajectory gradient estimale, as is done in PPO (Schulman et al.] 2017)). However, we note that the overall structure of many on-policy algorithms remains the same: use the current policy sample trajectories the environment; use the rewards, actions, and states along these trajectories to compute variable of interest x finally average the gradients of these sampled random variables to estimate Vg.((().
4 Multi-Fidelity Policy Gradients
This section introduces our MFPG framework, a mechanism to draw correlated trajectories from multi- fidelity environments, and the corresponding multi-fidelity REINFORCE algorithm.
Iteration notation. We consider an iterative policy gradient procedure with parameters 6g E Rd iteration & E Z+, and write Te, (a for the corresponding policy. When discussing single iteration and fixed parameter vector, all quantities are understood to be evaluated at that iteration, and we omit the subscript k for readability, writing simply 0 and тæ. All subsequent expressions that omit the iteration index should therefore be interpreted as taken at an arbitrary but fixed iteration k and hence apply without loss of generality to every iteration of the algorithm.
Multi-fidelity policy gradient estimators via control variates. Because our objective is to optimize the policy performance in the high-fidelity environment MM, during each step of the policy gradient algo- rithm must estimate (o from a potentially limited number N of sampled high-fidelity trajectories hh In data-scarce settings, existing policy gradient methods can face the challenge of high vari- ance of the gradient estimates (Greensmith al.) 2004). We aim to reduce the estimation variance for the PGs. In this work, assume that may also sample relatively large number N >> N of trajectories 7 from the low-fidelity environment M'. We use these low-fidelity samples to construct so-called control variate - correlated auxiliary random variable whose expected value μ²' := (r [X77] is known. Note that X" is obtained by applying the same trajectory functional (-) to trajectories 7' sampled from the low-fidelity environment (Te) under ma. We then use the control variates technique (Nelson 1987 to construct unbiased, reduced-variande estimator for (oEx">Aee<m0|XX!!
Specifically, we construct a new random variable (c) == X + c(X - -n") with coefficient c R. The next lemma formalizes the unbiasedness and pariance-reductiin properties of this construction; its proof follows standard control variate arguments and is provided in Section
Lemma 1 (Unbiascdness and variance reduction of the multi-fidelity control variate estimator). Let X and X be the high- and low-fidelity random variables defined above, and let μ² == (r) For any coefficient e R, define
Zeo (c) X To c(X -
(1)
Then, (c) is unbiased with respect to the high-fidelity random variable: E[Zn0 (c) E[x23).

---

Published in Transactions on Machine Learning Research (02/2026)
Moreover, by choosing the coefficient
Van(X*
c' = C,, xx)) p( ph x
Var(Z=((c)) = (1- Var Var(X;;), Var(X
Var(X
(2)
where pl:,.)) is the Pearson correlation coefficient between the two random variables. In particular, whenever MXahh X²P) + 0, the multi-fidellly control variale eslimalor z (c*) has strielly smaller variance than X alone.
We estimate s' using the N low-fidelity trajectory samples, and we provide a method to jointly sam- plc correlated values X and X (described below). In the remainder of the paper, we take Z == (c)) as the ideal fuulli-iiellity random variable. In practice e* is unknown and must be esti- mated from samples. We estimate e* using the sampled trajectories from M and M reusing sam- ples from previous gradient steps by maintaining moving-average statistics. Specifically, at iteration k, given the current policy Tea. first compute batch estimates for correlation phatchh (X""* X.")) and
standard deviations V Varx (x"**, V Var batch (x"7)) using newly sampled trajectorics at iteration k.
-- batch
Let P T0--1 Xx1" "()=")); V Vara- (X, - ), andy (ixi-1((,111))) denote the rumning estimates from the previous iteration. With moving average coefficient 7/ma E (0, 1), we update
Pa(x x=))) = Umaak
(x
Tak m +)+ (1 Thna) Ph
baatch
(X To
x )
V Vara (arr((Xaa" (x = Tmm V Vark-1 (X th 1) + (1 Mma))
Var
batch
(X TBN
(3)
Vara (X"+))= 7ma (aiis-(("" (x (1 7ma))
Vark
batch
(x (x50))
7'
We then form an estimate of the optimal control-variate coefficient as
E (x x V Vara (x,7*)
(4)
Varn (X
and, when the iteration index is clear, we simply write 8*
At every policy gradient step k given the current policy no, the proposed MFPG framework thus procoeds as follows: 1) Use policy na, to sample No correlated trajectories from M and M', well as No additional trajectories from M'. 2) Use the sampled trajectories to compute estimates Alg G. and the correlated values of the random variables X and x 3) Compute the sampled values of = 4) Use the samples of z to compute an unbiased, aeduced-varianee estimate of the policy gradient Vo, =
L Σ. Ve, [x"* Tep + +x Tex -x pk)]]
NW
Sampling correlated trajectories from the multi-fidelity environments. Note that for the random variables X and X to be correlated, they must share an underlying probability space F, F.I). In other words, every outcome w E Ω from this probability space should uniquely define a high-fidelity 7" (w) and low-fidelity ('(w) trajectory, as well as the corresponding values of the random variables X (w) and (w). We emphasize that the global outcome w is introduced purely as an analytical device. In practice, sampling and coupling the joint outcome across the high- and low-fidelity MDPs is infeasible, since the stochasticity in environment transitions and rewards is typically not directly controllable. Accordingly, MFPG algorithm only couples the policy randomness and the initial-state randomness across fidelities, while treating transition and reward outcomes as independent, as we will describe below. A careful formulation of the joint probability space is nevertheless useful for conceptual clarity and for describing how to construct correlated trajectories in M and M under the same policy Ty.
Informally, when sampling trajectories from M and M', there are six sources of stochasticity: the stochas- ticity introduced by the policy no, that introduced by the initial state distribution Asy; the stochasticity
7

---

Published in Transactions on Machine Learning Research (02/2026)
introduced by the high-fidelity and low-fidelity transition dynamics pd and p', and finally the stochasticity introduced by the two reward functions Rh and R'. Note that the stochasticity introduced by wo is im- plemented by the agent, and is thus under the control of the algorithm in the sense that the same policy random outcome w may be used to generate actions from me in both M and M'. We do so via a policy reparameterizzttion trick (see below). Similarly, the algorithm may fix the initial states in the low-fidelity environment to match those observed from the high-fidelity samples. On the other hand, the transition dynamics and environment rewards are generated by independent sources of stochasticity in the different environments; do not assume, nor require, any shared transition reward outcome across fidelities.
We accordingly define the outcome set of the probability space as ft = As., x St x Upr x Lpp x S2 x S2pt. Here, each outcome ~ Uz,, defines particular shared initial state. Meanwhile, each policy outcome WT Ωhr corresponds to a sequence w wi w ....,,,,,) wT that dictates the random sequence of actions selected by the policy He in both environments. In practice, this policy outcome sequence is realized sequence of per-timestep action-noise samples used to reparameterize the policy's action sampling at each time step (see the discussion on policy reparameterization below). Conceptually, given the outcome wp at any timestep of a trajectory, the policy should deterministically output action of = < w²) ) in the high- fidelity environment, and action a; = TTys), (") in the low-fidelity environment. Note that this does not necessarily imply that af a due potential difference states st and st that stems from the different dynamics. Similarly, each outcome wPn E E2pt is sequence 4 = wh" with w..,, dictating the outcomes 81+1 p"(,, of wdth) of the stochastic transitions in the high-fidelity environment. However, unlike the policy outcome sequence wn E 12T, the transition outcome sequences wPth E Q and w Ω2, and the reward outcome sequences R E R and Re E R are not shared between the high and low fidelity environments. Thus, only the initial-stale outcomes w and the policy outcomes wn are coupled across fidelities in our algorithm, while the transition and reward outcomes wPt w whth w remain independent. ,ht
Formulating these separate outcome sets is conceptually helpful. However, practically, we only explicitly sample values for the policy outcomes wp,wT wi, (action-noise samples) in our implementation, which describe next.
Algorithm 1: Correlated trajectory sampling.
Input: Current policy ne; multi-fidelity environments M and M'
Output: Sampled trajectories (7/1 N/ T 1 TrajectoryList + EmptyList
for i (1, ,N* do
wß ... ~ NompleeettionnoisseSSepppo sa A +
for 1 (O...., --
1} do
at + to(sh w // reparameterizzttion trick
841 p at:) r RD (s}: ai st-1)
for / (0, (0...., .T 1} do
at + no(s) wi²) =8) // reparameterizzatoo trick
t4 p'(110/,
of ~ R² R2(s// ob. sb41) ......, s
14
The s/. a nit
15
Tf + sb. ab. ro.....s.. 5f
16
TrajectoryList.aat append (
return TrajectoryList
8

---

Published in Transactions on Machine Learning Research (02/2026)
Correlated action sampling via policy distribution reparamcterization. In order to use the sam- pled outcomes W/ to deterministically select an action under the parameterized policy To, we implement a technique inspired by the so-called reparemeteriaation trick used in variational autoencoders (Kingma, 2013). In particular, in contimuous action spaces, we draw w/ ~ (N(, 1), and the policy (o()),o))) is trained to outpul a stale-dependent mean and standard deviation which are used to transform wi into an action 0g. Meanwhile, in discrete action spaces, one can draw wf ~ Uniform((, 1) and apply the Gumbel-Max trick (Huijben al. 2022) to sample ar according to the state-dependent probability distribution defined by the policy (s)
To summarize, Algorithm 1 outlines the sampling procodure. First, we use the pre-sampled policy action- noise sequence to roll out a high-fidelity trajectory 7", Next, the initial state in the low-fidelity environment is fixed to match the initial state of 7 and thereby effectively enforcing a common w Finally, we reuse the same sequence of action-noise samples to generate the low-fidelity trajectory r!.
Defining multi-fidelity variant of the REINFORCE algorithm. To this point, we have defined a mechanism for sampling correlated trajectories 7h and 7' from multi-fidelity environments M and M', as well as a framework for using said trajectories to construct reducod-variance estimators of the policy gradient from meeeetory-deeendent variables X and X=e With these elements of the MFPG framework in place, we may implement multi-fidelity variant of the REINFORCE algorithm. We note that, in principle, MFPG can also be instantiated with other on-policy policy gradient methods, by replacing the way in which the value of X is computed from sampled trajectories. In this work, we focus on multi-fidelity REINFORCE in order to case theoretical analysis, leaving extensions to other algorithms for future work.
We implement & multi-fidelity variant of the REINFORCE algorithm WWillaass 1988 1992 by defining X := IT G2 log (e(a; (st), as described above. Additionally, to match the commonly used variance- reduced variant of the REINFORCE algorithm (Peters & Schaal 2006), we subtract state values estimated by a value network from the Monte Carlo returns, i..,, X.5 == + +I (G, - Vø(ss)) log ( = 1 Er-1 Ag(st, az) log (s))))))) Here, Vo(*t), Ag(st, at) denote value and advantage functions estimated from samples. We sample trajectories from the high- and low-fidelity environments to compute Monte Carlo returns G.; and C,,, respectively. We then use shared value function V, learned from high-fidelity samples (s) af r 11 to compute state-value baselines that are subtracted from both G²₁² and G². This is an implementation choice that simplifies training; moreover, subtracting any state-dependent baseline does not change the expectation of the REINFORCE gradient estimator (Williams) 1992). Alternatively, one can train separate value functions for the high- and low fideiiility Monte Carlo returns. Algorithm 2 in Section summarizes the MFPG REINFORCE, algorithm.
T
A
5 Theoretical Analysis of Multi-Fidelity Policy Gradients
We now analyze the convergence of multi-fidelity policy gradients for the REINFORCE algorithm. We employ the following assumptions on the high and low-fidelity environment MDPs Mh and the parametrized policy ne, which are standard in reinforcement learning literature.
Assumption 1 (Bounded Rewards). The high- and low-fidelity reward functions Rhh It are bounded, i.e., there exists UR. Uk > 0 such that |R. a)l I U RR < Uk V (s) E S x A.
Assumption 2 (Differentiable Policy). The policy function Ty is differentiable with respect to I everywhere, and the score function V, log To(as)) exists everywhere.
Assumption 3 (Bounded Score Functions). The score function V log To is bounded, i.e., there exists Bg > 0 such that V [o()s))]] < Ba, A 0 E Rd, a) E x A.
Assumption 4 (Lipschitz Score Functions). The score function Volog To is Le-Lipschitz, i.c., | V log ne, (a|s) - log Te, [a|s)] < Le [0 - 02ll, V 01, 02 E Rd (s, A.
Assumption is common the policy gradient literaturr (Zhang et al. 2020 Assumptions to are also common in prior works investigating the convergence of various policy gradient Particularly, the commonly
Bhatnagar et al., 2007).
methods (Papini et al. 2018 2022 Pirotta et al. 2015 Zhang et al., 2020
9

---

Published in Transactions on Machine Learning Research (02/2026)
used Gaussian and Boltzmann parameterized policies are known to satisfy these assumptions Zhang et al, 2020j. We also make the following assumption on the high-fidelity policy gradient estimate.
Assumption 5 (High-Fidellty Policy Gradient Estimator). The high-fidelity policy gradient estimator
Va is unbiased and has bounded variance, i.e., E [VX = VeJ(O), and there exists o > 0 such
that IE |VoX, VoJ(()) 00.
Assumption| ensures that the high-fidelity policy gradient estimator is a "good" estimator, in the sense that the high-fidelity policy gradient would have learned a good policy had unrestricted access to high-fidelity data. Such an assumption required formally show convergence (see, e-g., Papini et al. (2018)) ) We note that Assumption 5 is an assumption on the underlying high-fidelity poliey gradient estimator; we do not require any policy gradient estimates computed from finite samples to be exact. In particular, the classic Monte Carlo REINFORCE policy gradient estimator considered in this work satisfies this requirement (Sut- lon et al. 1999.. Extending our analysis to actor critic---ie estimators is nontrivial and is discussed as a limitation and direction for future work in Section
Finally, in order for the proposed approach to be effective, there must be nonzero correlation between random variables computed from the high- and low-fidelity environments, i.e., the coefficient c* in Eq. 2 is nonzero. Hence, we make the following assumption.
Assumption 6 (Nonzero Correlation). The Pearson correlation coefficient between the high- and low-fidelity
random variables is nonzero: there exists p (0, 1] such that Cov (X"e XT?) / / Var (XT)) Var (x7+) Z
P, A e Rd
Assumption can be interpreted as comment on the quality of the low-fidelity estimator: while the low- fidelity estimator may be inaccurate and have return values that are incomparable to the high-fidelity due to different transition dynamics in the two environments, the assumption ensures that the lower fidelity still preserves some meaningful connection with the higher fidelity. This connection is captured by the correlation between the two fidelitics.
Under these assumptions, we establish that the variance-reduced MFPG iterates lead to a faster decrease in policy gradient norms in comparison to the high-fidelity-only iterates, when employed for finite horizon REINFORCE algorithm. Further, we establish the global asymptotic convergence of the MFPG to first- order stationary point of the high-fidelity objective function. The proof is presented in Section C
Theorem 1. Under Assumplions to 6 let (0MFPG) JEEE and (0k be the sequences of policy poram- eters generated by running the REINFORCE algorithm with MFPG and high-fidelity-onke policy gradient estimalors, respectively. Then, for a problem with time horizon T, after N ittrations, with slepsizes for both iterates taken to be a sequence og satisfying Ex ag = xc, Et a < co, the policy gradient (() evaluated the high-fidelity (target) environment is bounded as follows:
min E [IVJ(ONFFO)]] -((0) - ((1)) + +1 (1-p2) KE[N)
d a
E
2
IN
a?
LT
akk
where LT is the Lipschitz constant of the high-fidelity policy gradient, established in Lemma g Moreover, we recover the corresponding rate for the high-fidelity only policy iterates (oke by substituting = O. Finally, we have that limg-soc [[[ [[] = almost surely.
Proof Sketch. The non-asymplotic bounds leverage the Lipschitzness of the high-fidelity-only policy gra- dient estimator, which establish in Lemma in Section C while the asymptotic convergence proof utilizes the supermartingale convergence theorem (Robbins & Siegmund 1971) as follows. We first show that the norm of the MFPG estimator is bounded, then construct specific non-negative auxiliary random variable that we show is a supermartingale and apply the supermartingale theorem to establish the result. The complete proof is presented in Section
Theorem | establishes that the minimum (high-fidelity) policy gradient norm after k iterations will be smaller for the MFPG REINFORCE, algorithm compared to the corresponding high-fidelity-only algorithm.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Moreover, the improvement that the MFPG algorithm brings depends on how correlated the high and low- fidelity encironments are. Note that, as k increases, the right side of the inequality in Theorem I diminishes to 0, due to the stepsize conditions. Further, the theorem implies that, asymptotically, the MFPG estimator finds a policy that satisfies the first-order optimality conditions for the high-fidelity objective.
6 Experiments
This section empirically evaluates the proposed MFPG algorithm in a suite of simulated benchmark RL tasks. In high- high-fidelity data-scarce regimes, the experimental results demonstrate the following key insights
(II, Variance reduction) When the multi-fidelity dynamics gap is mild, MFPG can substantially reduce the variance of PG estimates compared standard variance reduction via state-value baseline subtraction.
(I2, Reliability and robustness) MFPG provides a reliable and robust way to exploit low-fidelity data. In our baseline comparisons in Section 6.3 for scenarios where low-fidelity data are neutral or beneficial (Hopper-gravity, Walker2d-frictionl and dynamics gaps are mild to moderate (0.5x to 2.0x), MFPG is, among the evaluated off-dynamics RL and low-fdeelty--nll approaches, the only method that consistently and statistically siguificantly improves the mean performance over a baseline trained solely on high-fidelity data in 8 out of 8 scenarios. When low-fidelity data become harmfull, MFPG exhibits the strongest robustness against performance degradation among the evaluated off-dynamics RL and low- uww-idelity--nly methods. Strong off dff-minasscs RL methods tend to exploit low-fidelity data more aggressively, and can achieve higher performance than MFPG times (e.g., in Walker2d) but can also fail substantially more severely (e.g., in HalfCheetah).
(I3, Misspecifted reward) MFPG can remain effective even when low-fidelity samples exhibit reward misspecification, e.g., when the low-fidelity reward is the negative (anti-correlated) version of the high-fieelity reward.
6.1 Experiment Setup
Task settings. We evaluate our approach the off-dynamics RL benchmark introduced by Lyu et al. (2024b), focusing on three standard MuJoCo tasks (Todorov et al. 2012): (i) Hopper-v3, (ii) Walker2d-v3, and (ii) HalfCheetah-v3. In each task, the low-fidelity environment corresponds to the original dynamics, while the high-fidelity variants include changes in gravity or friction. Gravity and friction are varied across five levels (0.5x,0.3) 0.8x, 1.2x, 1.0x, 5.0x]; the intermediate 0.8x and 1.2x levels are not part of the original benchnark but are introduced here examine settings with mild dynamics shifts. total, the lask set spans 15 distinct settings. In Sections 6.3 to 6.5 we evaluate each setting with 20 random seeds per method.
We provide additional results 24 further task settings in Section E evaluated with 5 random seeds per method. Due to the small number of seeds, these additional results are primarily used to illustrate the consistency of qualitative trends, and we do not draw formal statistical conclusions from them.
We note that the benchmark (Lyu et al. 2024b) also includes the Ant-v3 environment. Unlike the other three tasks, the observation space and reward lunction in Ant-v3 depend explicitly on contact forces, which cannot be reliably reset to user-specified values, as required by our MFPG sampling scheme. Due to this difficulty in matching initial states across fidelities, we focus our evaluations on the other three tasks described above.
Baselines. We compare MFPG against the following baselines: High-Fidelity Only, which trains solely on the limited high-fidelity samples; Low-Fidelity Only (100x), which trains on abundant low-fidelity data (100x more samples) and is evaluated in the high-fidelity environment; DARC (Eysenbach et al.). which learns domain classifiers to estimate dynamics gaps for the sampled low- fow-fidlitty transitions and uses the estimated gaps to augment Jow-fidelity rewards; and PAR (Lyu et al., (024a), which augments low- fidelity rewards using dynamics mismatch estimated in a learned latent space. Please refer to Section F for more detailed description of the baselines.
Code: https://xinjie- www.iiiiieeiii.itttiee. wtt.:i/-b..o///gg...1...
11

---

Published in Transactions on Machine Learning Research (02/2026)
We instantiate the MFPG framework with the on-policy policy gradient algorithm REINFORCE (Wiliamms 1992] with state value subtraction. DARC (Eysenbach et al.) and PAR (Lyu et al. 2024a were originally demonstrated with the ofl-po soft actor-critic algorithm (Haarnoja et al. 2018 To isolate the effect
of their multi-fidelity mechanisms from the backbone algorithm and enable fair comparison, we adapt both PAR and DARC to the REINFORCE backbone. We include our code in the supplementary materials and will release the project as open source upon acceptance.
Experiment regime. We evaluate MFPG in a regime where high-fidelity samples are scarce, reflecting many real-world applications such as robot learning and molccule design. To simulate this sctting, wo restrict all methods to a buffer of 100 high-fidelity samples per policy update. Multi-fidelity methods may supplement this limited buffer with up to 100x additional low-fidelity samples per update. Training for all methods is conducted for up 1M high-fidelity environment steps.
Hyperparameter configuration. Given the restriction on high-fidelity samples per policy update, we first tune hyperparameters for the High-Fidelity Only REINFORCE bascline (with state-value basclines subtracted). This luned configuration serves as the common backbone for all multi-fidelityy methods (MFPG, DARC, and PAR), in line with standard practice in the off-dynamics RL literature (Xu al. 2023 Lyu al, 2024a). Their method-specific hyperparameters, including the number of low-fidelity samples for DARC and PAR, are tuned separately. The Low-Fidelity Only (100x) baseline is implemented as direct ablation of the multi-fidelity methods. For further details hyperparameter setup, see Section F Overall, our tuning protocol grants the tested multi-fidelity baselines more tuning than MFPG, underscoring the simplicity and minimal tuning overhead of MFPG.
Notably, in our tuning process, PAR is particularly sensitive to hyperparameters; cf. Fig. 18 in Section Following (Lyu et al.) 2224a)), we adopt separate configurations per task for PAR, whereas all other methods use single configuration across settings. MFPG has the fewest hyperparameters among all the tested multi-fidelity methods. We tuned only the exponential moving- weight Mms in Eq. 3 and found that MFPG's is median performance consistently exceeded that of the High-Fidelity Only bascline across the tested values of 7ma, even though the absolute performance metrics varied with 7mma-
Result reporting. For the variance reduction study in Section 6.2 we run random seeds per method, this experiment already aggregates large number of gradient estimates within each run and is primarily intended to illustrate variance trends. For the experiments in Sections 6.3 to 6.5 we run 20 random seeds per method and task setting.
Performance evaluated in the high-fidelity environment every 2000 high- steps, using 10 evaluation episodes with a deterministic policy each time. We adopt two primary metrics: (i) the final evaluation return, defined as the return averaged over the last 20 evaluation steps, and (ii) area under curve (AUC), the area under the evaluation-reluu high-fidelity step axis using the composite trapezoidal rule. AUC captures a method's accumulated return performance (Stadic et al.] 2015; Osband et al. 2016 Hessel et al.] 2018). Finally, to assess robustness, we also report the number of performance collapses, defined as cases where the median of a method falls below 50% of the median value of the corresponding High-Fidelity Only baseline.
curve across training, estimated by integrating evaluation return along the
Uncertainty and significance. Where reported, we compute two-sided 95% nonparametric bootstrap confidence intervals using R = 10,000 resamples (sampling with replacement over seeds). We report point estimates as sample means. In the baseline comparisons in Section 6.3 we bootstrap the mean difference of each method compared to High-Fidelity Only, i.c., A = mcanmethod Fialiiih-Fidllltyy Only, by independently resampling seeds within each method and recomputing A for each resample. We declare an improvement statistically significant at 0.05 when the 95% bootstrap confidence interval for A excludes zero.
12

---

Published in Transactions on Machine Learning Research (02/2026)
High-Fid, Only (batch 100)
High-Fid. Only (batch 100, value subt.)
5-10
MEPG (oum, batch 100)
Variance
10 5-100
MFPG (oum, batch 100, value subt.)
High-Fid. Only (batch 1000)
10
High-Fid. Only (batch 1000, value subt)
High-Fid. Only (batch 3000)
High-Fid. Only (batch 3000, value subt)
High-Fidelity Enw. Steps
leb
Var. Percent.
100
50
0.2
0.4
0.6
0.8
1.0
High-Fidelity Env. Steps
1e6
Figure 2: Variance of policy gradient estimates for MFPG versus variants of the High-Fidelity Only baseline on Hopper-v3 (top), and the percentage of MFPG variance relative to singlc-fidelity counterparts of the same batch size (bottom). Solid lines denote the median across seeds, and shaded regions indicate the corresponding maximum and minimum values. In high-fidelity data-scarce regimes with mild dynamics gaps, MFPG generally exhibits far lower PG variance than the singlo-fieelity counterparts.
6.2 Empirical Variance Reduction
This experiment supports our key insight I1: When high-fidelity samples are scarce and the dynamics gap between environments is mild, MFPG substantially reduce the variance of policy gradient estimates compared to standard variance reduction via baseline subtraction.
We use Hopper task with 1.2x friction as an example. Figure 2 (top) shows the variance of PG estimates from MFPG versus a few variants of the High-Fidelity Only baseline (more specifically, we plot the variances of the scalar quantities Z and X before differentiation with respect to policy parameters). The plotted High-Fidelity Only variants include a baseline that has access to the same limited number of high-fidelity samples per policy update as MFPG (i.e., the same batch size), baselines with more high-fidelity samples (a batch size of 1000 and 3000), and variants of these High-Fidclity Only basclines that also state-valuc function subtraction in effort to reduce variance.
We train a policy using the single-fidelity REINFORCE algorithm with state-value subtracted for million steps and save the trained policy at 18 different checkpoints. For each of these saved policies, we collect 200 batches of both high- and low- cow-iiieitty data (with low-fidelity data collected at 100x the amount of high- fidelity data ), where the size of each high-fidelity batch, i.c., the amount of high-fidelity data, varies between approaches as described above. We then record the empirical variance of the policy gradient estimates from these 200 batches, for each checkpointed policy. We repeat. this experiment for 5 random seeds and report aggregate statistics in Figure 2 (top) where each line is based on 21600 batches of policy gradient estimates. Solid lines denote medians and shaded regions (which largely overlap the median lines) indicate the range between maxima and minima. across the 5 seeds. Figure 2 (bottom) reports the ratio of median variance of policy gradient estimators for two variants of MFPG (with and without value-funde function subtraction). measured relative to their single-fidelity counterparts under the same batch size.
As shown in Figure in data-scarce regimes, MFPG can substantially reduce the variance of policy gradient estimators compared to single-fidelity baselines at most training checkpoints. This variance reduction is generally far greater than that achieved by standard state-value subtraction, narrowing the gap to baselines that rely on significantly more high-fidelity samples (e.g., 10x).
6.3 Baseline Comparison Across Dynamics Variation Types and Levels
This section reports the evaluation of MFPG against baseline methods under the task settings described in Section 6.1 and supports our key insight I2.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Off-Dynamics Comparisons Evaluation (Relative High-Fidelity Only)
MIPG
DARC
PAR
Low--idelltt 000x)
Final FinaIreture return
150
0.8x
1.2x Hopper-gravity gravity
2.0x
Final return
600
300
WalkerZd friction
1500
Final
11500
0.5x
0.8x
1.2v
2.0x
5.0x
HalfCheetah friction
Figure Mean difference final evaluation return for each method relative to High-Fidelity Only, com- puted as A meanmmethod HieaHHig-Fidelle Only (0 corresponds to High-Fidelity Only). Bars show the bootstrap point estimate of A, and error bars denote two-sided 95% bootstrap confidence intervals for A. When low-fidelity data are neutral or beneficial (rows 1-2) and dynamics gaps are mild moderate (0.5x to 2.0x), MFPG is the only method among the evaluated approaches that consistently attains statistically sig- nificant improvements over High-Fidelity Only. Strong Off-dynamics RL baselines tend to exploit low-fidelity data aggressively, which can occasionally yield high performance (e.g., in Walker2d) but can also degrade performance substantially more severely than MFPG when low-fidelity data become harmful; ef. row 3 and
Fig.
Off- Off-Dynamics Task Comparisons: AUC (Relative to High-Fidelity Only)
MIPG
=
DARC
=
FAR
= Lou-Fidellty Only (1860
Minu
Puldi
AUC
280
0.5x
0.8x
1.xx HallCheetah-friction
2.0x
5.0k
Figure 4: Mean difference in AUC for each method in HalfCheetah- friction relative to High-Fidelity Only. computed analogously to Fig. 3 (0 corresponds to High-Fidelity Only). Bars show the bootstrap point estimate of the mean difference, and error bars denote two-sided 95% bootstrap confidence intervals. When low-fidelity data become harmful, MFPG exhibits the strongest. robustness to performance degradation among the evaluated methods.
Fig. 3 presents the mean difference in final evaluation return of each method relative to the High-Fidelitg Only baseline, measured as the difference in mean final evaluation return = meanmeethod mean (so corresponds to High-Fidelity Only), on 15 representative task settings, with error bars indicating two- sided 95% bootstrap confidence intervals for A. Each row corresponds to one task, with the high-fidelity enviromment varying across multiple dynamics levels. For the simpler Hopper-gravity and Walker2d-friction tasks, abundant low-fidelity data tend to be neutral or beneficial and training is relatively stable, the final evaluation return and AUC metrics are generally consistent; thus, report only the final evaluation return. In contrast, the HalfCheotab-frictioon task is more challenging and low-fidelity data tend to be harmful: training exhibits greater fluctuations that occasionally reduce reward performance (baselines fluctuale far more substantially than MFPG). To capture accumulated learning performance more comprehensively in these settings, we report both the final return (Fig. row 3) and the AUC metric (Fig. 4 We compute and visualize the AUC results analogously, i.c., as differences relative to Hfgh-Fidelity Only with two-sided 95% bootstrap confidence intervals.
Only
14

---

Published in Transactions on Machine Learning Research (02/2026)
Hopper and Walker2d tasks. As shown in Fig. 3 (rows 1-2), in these two tasks, abundant low-fidelity data generally tend to be neutral or beneficial: across all settings, the confidence intervals of the Low- Fidelity Only baseline are never strictly below O. MFPG is the only evaluated method that reliably and consistently achieves statistically significant improvements when the dynamics gap is mild to moderate (0.5x to 2.0x) iee, with error bars strictly above O in all 8/8 such scenarios by leveraging low-fidelity data and cross-fidelity correlation to complement the limited high-fidelity data.
No other off-dynamics RL or ool-ii bascline achieves this level of consistency. For instance, in Hopper-v3 with a 0.5x gravity shift, the Low-Fidelity Only baseline shows neutral performance relative to High-Fidelity Only. Consequently, the evaluated off-dynamics RL basclines fail to significantly improve over High-Fidelity Only, whereas MFPG effectively combines high- and low-fidelity data, leveraging cross-fidelity correlations to achieve a statistically significant performance gain over the High-Fidelity Only baseline.
When low-fidelity data tend to be strongly beneficial, c.g., in Hopper-v3 with a 1.2x gravity shift and Walker2d-v3 with 0.8x -20x friction shifts, the evaluated off-dynamics RL baselines aggressively exploit the abundant low-fidelity data to boost performance, achieving significantly higher mean returns than High- Fidelity Only. By contrast, MFPG behaves more cautiously, providing reliable and consistent yet still statistically caaaffcant-improeemmont over High-Fidelity Only.
We emphasize that this behavior of MFPG is by design: MFPG uses low-fidelity data solely as a variance- reduction tool, deliberately trading potentially aggressive (but risky) gains for reliability. This design choice underlies the consistency observed here and the strongest robustness that MFPG exhibits relative to the other evaluated off-dynamics RL methods in the HallCheetah-v3 task presented below.
In extreme-shift scenarios, i.e., 5.0x dynamics shifts, cross-fidelity correlation drops sharply and MFPG performance becomes effectively neutral relative to High-Fidelity Only.
Another interesting observation is that, even when it improves over High-Fidelity Only, the Low-Fidelity Only baseline tends to exhibit high cross-seed variance, as reflected by its long error bars in Hopper-v3 and Walker2d-v3 with 0.8x-2.0x dynamics shifts the box plots shown Figs. [4 and 15 Moreover, whether Low-Fidelity Only significantly improves over High-Fidelity Only does not correlate monotonically with the dynamics gap; for example, low-fidelity data appear more beneficial in Hopper-v3 with 2.0x gravity shift than with a 0.5x shift. This highlights both the inherent difficulty of predicting when zero-shot transfer will be effective in practice and the importance of having reliable multi-fidelity approaches.
HalfChectah task. As shown in Fig. 3 (row 3) and Fig. in this task, low-fidelity data tend to be harmful: DARC and Low-Fidelity Only consistently and significantly degrade mean final return performance over High-Fidelity Only across mild-to-moderate gaps (0.5x to 2.0x), whereas PAR still manages to improve significantly over High-Fidelity Only and achieves the highest performance among the evaluated methods in these scenarios. However, PAR's performance gains come at the cost of computationally expensive, per-task hyperparameter tuning. We empirically observed that PAR is highly sensitive to the reward-augmentation weight 3; see Fig. 18 in Scction
F Even with such per-task tuning, PAR's performance often exhibits large cross-seed variance with low minima. (ef. 0.5x friction in Fig. 15 and several additional cases in Figs. 16 and 17p.
By contrast, MFPG does not require such expensive, careful tuning: thanks to its algorithmic simplicity, we only tuned the exponential moving-average weight. Mmaa in Eq. (3) during our experiments. Across mild-to- moderate gaps, MFPG is robust to harmful low-fidelity data and remains effectively neutral with respect to High-Fidelity Only.
At an extreme dynamics shift, i.e., 5.0x friction, the performance of DARC, PAR, Low-Fidelity Only all drops sharply, with the confidence intervals for both mean final return and AUC for below 0. In this scenario, we observe that the mean return curve of MFPG (cf. Fig. 4 initially closely tracks that of High-Fidelity Only before exhibiting some instability, resulting in a lower mean final return but relatively strong accumu- lated performance (i.e., AUC). By contrast, all other baselines struggle to achieve meaningful performance throughout training, suggesting that MFPG is the most robust among the evaluated off-dynamics RL and ooly methods in this setting.
15

---

Published in Transactions on Machine Learning Research (02/2026)
The observed instability of MFPG may stem from noise in the estimated control-variate coefficient é", and designing more reliable estimation schemes constitutes an interesting direction for future work.
Robustness. The trends in Figs. 3 and f indicate that MFPG is the most robust among the evaluated off-dynamics RL and low-fidelity-oni numbers of performance colla pses (final return AUC) are: MFPG (1 / 0), PAR
methods. Quantitatively, across the 15 scenarios in Figs. B and 1 the
1), Low- FideFiy Only (100x) (5 3), and DARC (5 / 3). Across the additional 24 settings in Section L the corresponding
collapse counts (final return / AUC) are: MFPG (1 0), PAR (3 1), Low-Fidelity Only (100x) (8 7), and DARC (7) 55.
Taken together, these results support our key insight 12 regarding the reliability and robustness of MFPG.
MFPG performance vs. correlation. We examine one task in detail to illustrate the qualitative behavior of MFPG under varying dynamics gaps. Figure [ presents evaluation-return curves of MFPG and High- Fidclity Only the Hopper task with three levels of gravity variation (0.8x, 2.0x, and 5.0x), along with the estimated Pearson correlation coefficients between high- and low-fidelity policy gradient losses throughout training. Solid lines indicate bootstrap point estimates (means), and shaded regions denote two-sided 95% bootstrap confidence intervals. As the dynamics shift incrcases, the cross-fidelity correlation decreases (high for 0.8x, moderate for 2.0x, and near-zero for 5.0x). Consistent with this, the gap between MFPG's mean return and that of High-Fidelity Only is largest when the correlation is high (0.8x) and smaller when the correlation is moderate (2.0x); in both cases, MFPG trends above High-Fidelity Only for much of training, although the bootstrap confidence intervals overlap during some portions. At the extreme 5.0x shif-wwhre the correlation collapses- the two methods exhibit very similar learning curves, with largely overlapping bootstrap confidence intervals. Figure 5 shows both algorithms' performance for completeness. Figure further reports the mean difference in evaluation return between MFPG and High-Fidelity Only, computed as A meallmethod - -iedellgh- Only ( A = 0 indicates no difference). For mild (0.8x) and moderate (2.0x) dynamics gaps, the confidence intervals are often above zero, suggesting improved performance, with larger gains typically observed in the mild-gap case. Under the extreme shift (5.0x), the confidence intervals largely span zero. The AUC results in Fig. 12 are consistent with these trends.
MPPG
High-Fidelity Only
400
605
600
1.4
0. grovity
Tesao Conelatur Colfiiciant
1.2
430
600
200
1.0
Teld
0.8
200
200
200
0.3
0.4
Steps
S.0
High Fidelly Envinonment Stepo
112
B4
0.6
0.5
1.0
Hiigh-hdeelty
82
Stepe
MighhRaariy Eastroment
04
dynamies shift
smaller (0.5x)
larger (5x)
Figure Evaluation return curves of MFPG versus the High-Fidelity Only baseline on Hopper-v3, cor- responding to Fig. 3 under mild, moderate, and large gravity variations (left to right: 0.8x, 2.0x, 5.0x), together with estimated Pearson correlation coefficients between high- and low- low-fidelity policy gradient losses. Solid lines indicate bootstrap point estimates (means), and shaded regions denote two-sided 95% bootstrap confidence intervals. Consistent with the correlation estimates, the gap between MFPG's mean return and that of High- Righliiy Only is more pronounced when cross-fidelity correlation is stronger.
6.4 Low-Fidelity Environment with a Misspecified Reward
This experiment designed to support our key insight I3, that when high-fidelity samples are scarce, MFPG can remain effective even in the presence of reward misspecification in the low-fidelity samples, e.g., when the low-fidelity reward is the negative (anti-correlated) version of the high-fidelity reward.
16

---

Published in Transactions on Machine Learning Research (02/2026)
As discussed in Section 4 long there is statistical relationship between the random variables of interest X and X (i.e., p²(x= "h; x²)) is non-negligible). the MFPG framework will reduce the variance of the policy gradient estimates (with respect to the high-fidelity environment).
To demonstrate this point, we examine situation in which the reward function in the low-fidelits envi- ronment the negative of that from the high-fidelity environment. Figure D reports the evaluation return the High-Fidelitty Only and Low-Fidelity Only baselines. Solid lines indicate
of MFPG in comparison bootstrap point estimates (means), and shaded regions denote two-sided 95% bootstrap confidence intervals. All approaches in this example use the plain REINFORCE algorithm without state-value subtraction. The core idea extends to the case with value-function subtraction, except that separate value functions must be learned for the multi-fidelity environments because of the misspecified reward.
As expected, the Low-Fidelity Only baseline is entirely ineffective under the drastically misspecified reward: instead of Iearning to move forward, the agent Icarns to end the episode as quickly as possible. Mcanwhile, the High-Fidelity Only baseline makes limited progress due to the scarcity of high-fidelity samples. In contrast, MFPG effectively combines the two data sources and achieves substantially better performance than both baselines.
Intuitively, although the values of X are entirely different from those of X they are negatively correlated in this example. The MFPG algorithm takes advantage of this relationship to compute a correction for the low-fidelity estimator fl, using only a small number of high-fidelity samples. Although this is an extreme example in the sense that the low- and high-fidelity tasks are polar opposites of each other, it highlights a useful feature of the MFPG framework: the low-fidelity rewards and dynamics might be very different from the target environment of interest (making direct sim-to-real transfer infeasible), and yet still provide useful information for multi-fidelity policy training.
These results support our key insight 13. Taken together, our experiments volidate all three key insights (31-I3).
MFPG
MF'G
Final return
800
High-Fidelity Only
150
Low-Fidelity Data Only (100x)
Reward
600
100
200
1000
2000
1e8
Samples Policy Update
200
AUC
01
+
sb
High-Fidelity Environment Steps
1eb
100
200
500
1000
2000
Figure 6: Evaluation return curves of MFPG, High-Fidelity Only, and Low-Fidelity Only on Hopper-v3, where the low-fidelity environ- ment misspecified with negated reward. Solid lines indicate bootstrap point estimates (means), and shaded regions denote two-sided 95% bootstrap confidence intervals. Although low-fidelity data alone is ineffective for train- ing, MFPG successfully leverages both fideli- ties achieve substantial improvements over High-Fidelity Only in high-fidelity data-scarce regimes.
Samples/ / Policy Update
Figure 7: Mean difference in final return (top) and AUC (bottom) between MFPG and High- Fidelity Only on Hopper-v3 with 1.2x fric- tion, across varying high-fidelity batch sizes (i.e., samples per update). Zero corresponds to HiighFFieelity Only. Error bars indicate two- sided 95% bootstrap confidence intervals for the mean difference. MFPG provides larger benefits over High-Fidelity Only in high-fidelity sample- scarce regimes.
17

---

Published in Transactions on Machine Learning Research (02/2026)
6.5 Additional Experiments
This section presents additional experiments that provide more detailed examination of different aspects of the MFPG framowork. For brevity in the main text, Section D presents additional ablation results.
MFPG performance with different amounts of high-fidelity samples. Our main evaluation in Section 6.3 focuses on the high-fidelity data-scarce regime, which reflects many real-world scenarios where collecting large quantities of high-fidelity samples for each policy update is prohibitively expensive (e.g., in robotics molecule design). This section studies how MFPG compares to the High-Fidelity Only bascline as the nunber of high-fidelity samples per policy update varies.
Figure 7 reports the mean difference in final return (top) and AUC (bottom) between MFPG and High- Fidclity Only in the Hopper task with 1.2x friction. Methods are trained for 1 million steps, comparing MFPG with 100x low-fidelity data against High-Fidelity Only across different. batch sizes (i.e., numbers of high-fidelity samples per update). Error bars indicate two-sided 95% bootstrap confidence intervals for the mean difference. Note that with fixed total number of high-fidelity environment steps, the number of gradient updates varies across batch sizes; consequently, the metries for High-Fidelin Only do not necessarily increase monotonically with larger batch sizes. In high-fidelity data-scarce regimes, e-g., batch sizes of 100 and 200, MFPG yields statistically significant improvements over High-Fidelity Only with relatively large mean differences, and the margin decreases as the batch size increases. When high-fidelity samples are abundant, e.g., batch sizes above 500, MFPG is largely neutral relative to High-Fidelity Only (though it still shows a small but statistically significant improvement at batch size of 2000). These results highlight that MFPG is most effective in high-fidelity data-scarce settings.
MPPG (2))
(20)
MFPG (100)
MPFG
(50c)
MPYG
MPPG (10k)
Final Retur
MFPGietan0.92) MFFCGetaa0.55)
MFPGGeta-0.97)
MFPG(etan0.99)
150
Final Return
150
Hopper 0.5x friction
Hopper 12x friction
1e8
2
Hopper 0.5x friction
Hopper 1.2x friction
AUC
AUC
2
1
Hopper 0.5x friction
Hopper 12x friction
Hopper 0.5x friction Hopper 1.2x friction
Figure 8: Mean difference in final return (top) and AUC (bottom) of MFPG with in- creasing amounts of low-fidelity data (light to dark orange) relative to High-Fidelity Only on Hopper-v3 with 0.5x and 1.2x fric- tion (0 corresponds to High-Fidelity Only). Error bars indicate two-sided 95% boot- strap confidence intervals for the mean dif- ferences. MFPG is relatively robust to the amount of low-fidelity data: across the lested range of low-fidelity sample sizes, we do not observe statistically significant degradation relative to High-Fidelity Only under our significance test, and MFPG is often statistically significantly better.
Figure 9: Mean difference in final return (lop) and AUC (bottom) of MFPG with varying expo- nential moving-average weights (7ma in Eq. (3) relative to High-Fidelity Only (0 corresponds to High-Fidelity Only) on Hopper-v3 with 0.5x and 1.2x friction. Error bars indicate two-sided 95% bootstrap confidence intervals for the mean differ- ences. Across the tested values of 7maa MFPG is often statistically significantly better than High- Fidelity Only, and we do not observe statistically significant degradation.
18

---

Published in Transactions on Machine Learning Research (02/2026)
MFPG performance with different amounts of low-fidelity samples. We assume throughout this work that low-fidelity samples are cheap to collect, and all methods are given access to an abundant budget of low-fidelity data. Consequently, optimizing the efficiency of low-fidelity sample usage (Cutler et al. 2015)) is not the focus of this work. Instead, this scction qualitatively examines how the performance of MFPG varies with different amounts of low-fidelity data.
In principle, the performance of MFPG is robust with respect to the amount of low-fidelity samples, so long the low-fidelity sample amount is sufficient to estimate the low-fidelity sample mean p'. Figure g reports mean differences in final return (top) and AUC (bottom) of MFPG relative to High-Fidelityy Only (0
corresponds to High-Fidelity Only) with increasing amounts of low-fidelity samples (indicated from light to dark orange) for estimating low-fidelity sample mean (a' in Fig. [p, in Hopper with 0.5x and 1.2x friction. Error bars indicate two-sided 95% bootstrap confidence intervals.
At a high level, MFPG is relatively robust to the amount of low-fidelity data: in our experiment, varying the low-fidelity sample amount over wide range does not lead to performance that is statistically significantly worse than Fidelityy Only in both settings, and MFPG is often statistically significantly better. When the dynamics gap is mild (e.g., 1.2x friction), for the AUC metric, increasing the number of low-fidelity samples tends to yield a higher bootstrap mean improvement in AUC over High-Fidelity Only. In particular, the confidence intervals for 100x and 200x low-fidelity data lie clearly above those for 2x low-fidelity data, suggesting a significantly faster learning process. However, under larger dynamics gaps, no clear monotonic trend is observed. Overall, additional low-fidelity samples are more beneficial in mild-gap settings.
Sensitivity to exponential moving-average weight. We further examine the sensitivity of MFPG to the choice of the exponential moving-average weight 7ma in Eq. (3). Figure 9 reports the mean differences in final return and AUC between MFPG (with different 7ma values) and High-Fidelity Only (0 corresponds to High-Fidelity Only) in the Hopper task under 0.5x and 1.2x friction. Error bars indicate two-sided 95% bootstrap confidence intervals for the mean differences. Overall, across the tested values of Mmaa, MFPG is often statistically significantly better than High-Fidelity Only, and we do not observe statistically significant degradation. The mean performance improvement of MFPG over High-Fidelity Only is generally more substantial in settings with smaller dynamics gaps consistent with the key insights from our baseline comparison in Section 6.3
7 Conclusions
We present a multi-fidelity policy gradient (MFPG) framework, which mixes small amount of poten- tially expensive high-fidelity data with a larger volume of cheap lower-fidelity data to construct unbiased, variance-reduced estimators for on-policy policy gradients. We instantiale this framework through a practi- cal, multi-fidelity variant of the standard REINFORCE. algorithm. Theoretically, we show that under stan- dard assumptions, the MFPG estimator guarantees asymptotic convergence of multi-fidelity REINFORCE to locally optimal policies (up to first order optimality conditions) in the target high-fidelity environment of interest. Moreover, whenever the statistical correlation between high- and low-fidelity samples is nontrivial, MFPG REINFORCE achieves faster finite-sample convergence than its single-fidelity counterpart. Empiri- cal evaluation of MFPG in high-fidelity data-scarce regimes across a suite of simulated robotics benchmark tasks with varied multi-fidelity transition dynamics highlights both its reliability and robustness. In our baseline comparison experiments, supported by a 20-seod statistical analysis, for scenarios where low-fidelity data are neutral or beneficial and dynamics gaps are mild to moderate, MFPG is, among the evaluated off off-dynamics RL and low-fidelity-onke approaches, the only method that consistently achieves statistically significant improvements in mean performance over baseline trained solely on high-fidelity data, When low-fidelity data become harmful, MFPG exhibits the strongest robustness against performance degradation among the evaluated methods. By contrast, strong off-dynamics RL methods tend to exploit low-fidelity data more aggressively, and can achieve high performance at times, but can also fail substantially more severely. Furthermore, an additional experiment in which the high- and low-fidelity environments are as- signed anti-correlated rewards demonstrates that MFPG can remain effective even in the presence of reward misspecification in the low-fidelity environment.
19

---

Published in Transactions on Machine Learning Research (02/2026)
In summary, the proposed framework offers reliable and robust paradigm for leveraging multi-fidelity data in reinforcement learning, providing promising directions for efficient sim-to-real transfer and principled approaches to managing the trade-off between policy performance and data collection costs.
Limitations & future work. Future rescarch spans several avenues. First, we study the classic REIN- FORCE algorithm as starting point to ease theoretical analysis. Extending the MFPG framework to modern RL inchhudin actor-critic, off-policy, and offline methods-raiee new questions about bias virs-nari trade-offs that merit deeper investigation. We defer a more detailed discussion of this point to Scction G
Second, MFPG relies on nontrivial statistical correlation between multi-fidelity samples to achieve variance reduction. Developing systematic techniques enhance correlation between high- and low-fidelity samples without introducing bias is critical for further performance gains.
Third, our analysis and implementation make several simplifying assumptions, and extending the MFPG framework to more general settings is of great importance. These extensions include (i) integrating data from an arbitrary number of environments, each simulating the target environment with a different fidelity level and cost. per sample, (ii) handling multi-fidelity MDPs with different state and action spaces, initial- state distributions, or discount factors, (iii) extending MFPG infinito-horizon MDPs, and (iv) devising mechanisms sample correlated trajectories even when low-fidelity simulators do not support resets to arbitrary user-specified initial states.
Finally, applying MFPG to real-world robotic systems represents an important step toward rcalizing its potential in practical sim-to-real transfer.
Broader Impact Statement
This work introduces a methodological contribution multi-fidelity policy gradients for improving the (high-fidelity) sample efficiency of reinforcement learning in settings with access to both high- and low-fidelity environments. We do not foresee immediate societal risks arising directly from this research. Potential down- stream applications may include robotics and autonomous systems, where increased efficiency can reduce computational and data collection costs.
Acknowledgments
We thank Haoran Xu, Mustafa Karabag, Brett Barkley, and Jacob Levy for their helpful discussions. This work was supported in part by the National Science Foundation (NSF) under Grants 2409535 and 2214939, in part by the Defense Advanced Research Projects Agency (DARPA) under the Transfer Learning from Impre- cise and Abstract Models to Autonomous Technologies (TIAMAT) program under grant number HR0011-24 9-0431, and in part by the DEVCOM Army Research Laboratory under Cooperative Agreement Numbers 2011NNF-23-2-011 and W911NF-25-220020 those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
The views and conclusions contained in this document are
References
Pieter Abbeel, Morgan Quigley, and Andrew Y Ng. Using inaccurate models in reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pp. 1-8, 2006.
Akash Agrawal and Christopher McComb. Adaptive learning of design strategies over non-hierarchical multi-fidelity models via policy alignment. orXiv preprint arX1t:2411.108,1, 2024.
Anonymous. Cross-domain offline policy adaptation via scloctive transition correction. In Submātted The Fourteenth International Conference on Learning Representations, 2025. URL htttps://ppurrevee..r forum? forum??ddMME:RRpJDWZ under review.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki. Meta reinforcement learning for sim-to- real domain adaptation. In 2020 IEFE international conference on robolics and automation (TCRA), pp- 2725 2731. IEEE, 2020.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 1127381-27394 2021.
Shalabh Bhatnagar, Mohammad Ghavamzadeh, Mark Lee, and Richard S Sutton. Incremental natural actor-critic algorithms. Advances in neural information processing systems, 20, 2007.
Sahil Bhola, Suraj Pawar, Prasanna Balaprakash, and Romit Maulik. Multi-fidelity reinforcement learning framework for shape optimization. Journal of Computational Physics, 482:112018, 2023. ISSN 0021-
9991. doi: https:///doi.oor//00// 10t./ 112018. URL atticllepp///002299133301111
com/science/
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8973 8979, 2019. doi: 10.1109/ICRA. 2019.8793789.
Ching-An Cheng, Xinyan Yan, and Byron Boots, Trajectory-wise control variates for variance reduction in policy gradient methods. In Conference on Robot Learning, pp. 1379-1394. PMLR, 2020.
Mark Cutler, Thomas J. Walsh, and Jonathan P. How. Real-world reinforcement learning via multifidelity simulators. IEEE Transactions Robotics, 31(3):655 :655-671, 2015. doi:
Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, and Hua Wei. A survey of sim-to-real methods in rl: Progross, prospects and challenges with foundation models. arXio preprint ar Xrv::22002 13187, 2025.
Peter Dayan. Reinforcement comparison. In Connectionist Models, pp. 45 51. Elsevier, 1991.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):444-419, 2022.
Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismaich. Advances in Neural Information Processing Systems, 0133977-9929 2020a.
Siddharth Desai, Haresh Karnan, Josiah P Hanna, Garrett Warnell, and Peter Stone. Stochastic grounded action transformation for robot learning in simulation. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 6106-6111. IEEE, 2020b.
Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhutdinov. Off- dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representations.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Raj Ghugare, Santiago Miret, Adriana Hugessen, Mariano Phielipp, and Glen Berseth. Searching for high- value molecules using reinforcement learning and transformers. arXiv preprint arXiv:2310 0222030,,,090,, 2023.
Will Grathwohl, Dami Choi, Yuhuai Wu, Geoff Roeder, and David Duvenaud. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. In International Conference on Learning Representations, 2018.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient esti- mates in reinforcement learning. Journal of Machine Learning Research, 5(Nov):1471-15333 2004.
Shixiang Gu, Timothy Tallicrap, Zoubin Ghahramani, Richard E. Turner, and Sergey Levine. Q-prop: Sample- efficient policy gradient with an off- aff-policy critic. In International Conference on Learning Representations, 2017.
Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, and Anqi Liu. Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation. Advances in Neural Information Processing Systems, 37:136326 136360, 2024.
Yihong Guo, Yu Yang, Pan Xu, and Auqi Liu. MOBODY: Model based off-dynamics offline reinforcement learning. arXiv preprint arXiv:2506. 2tf::2566008600,, 2025.
Tuomas Haarnoja, Aurick Zhou, Picter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pp. 1861-1870. Pmlr, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hassolt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jiawei Huang and Nan Jiang. From importance sampling to doubly robust policy gradient. In International Conference on Machine Learning, pp. 4434-4443. PMLR, 2020.
Tris AM Huijben, Wouter Kool, Max B Paulus, and Rund JG Van Sloun. A review the gumbel-max trick and its extensions for discrete stochasticity in machine learning. IEEE transactions on pattern analysis and machine intelligence, 45(2):1353-133 2022.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, (20(797)):982-20)
2023.
Sami Khairy and Prasanna Balaprakash. Multi-fidelity reinforcement learning with control variates. Nearo- computing, 597:127963, 2024.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiu:1412.6980 2014.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint #rXim:2110.06169 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances in neural information processing systems, 18:111991199) 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng, and Qiang Liu. Action-dependent control variates for policy optimization via stein identity. In International Conference on Learning Representations, 2018.
Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-aware reward angmentation in offline reinforcement learning. Xix preprint or Xim:2203. 06662, 2022.
Lennart Ljung. System identification. In Signal analysis and prediction, pp. 163 173. Springer, 1998.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Jiafei Lyu, Chenjia Bai, Jing- Wen Yang, Zongging Lu, and Xiu Li. Cross-domaie policy adaptation by capturing representation mismatch. In International Conference on Machine Learning, pp. PMLR, 2024a.
Jiafei Lyu, Kang Xu, Jiacheng Xu, Mengbei Yan, Jingwen Yang, Zongzhang Zhang, Chenjia Bai, Zongqing Lu, and Xiu Li. Odrl: benchmark for ofl-dynamics reinforcement learning. In The Thirby-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b. URL
Jiafei Lyu, Mengbei Yan, Zhongjian Qiao, Runze Liu, Xiaoteng Ma, Deheng Ye, Jing- Wen Yang, Zongqing Lu, and Xiu Li. Cross-domain oflline policy adaplation with optimal transport and dataset constraint. In The Thirteenth International Conference on Learning Representations, 2025.
Alonso Marco, Felix Berkenkamp, Philipp Hennig, Angela P Schoellig, Andreas Krause, Stefan Schaal, and Sebastian Trimpe. Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with bayesian optimization. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1557-1563. IEFE, 2017.
Alberto Maria Metelli, Matteo Papini, Nico Montali, and Marcello Restelli. Importance sampling techniques for policy optimization. Journal of Machine Learning Research, 21(141):1-75, 2020.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132):1 62, 2020.
Barry L. Nelson. On control variate estimators. Computers & Operalions Research, 14(3):219 225, 1987.
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When trust your sim- ulator: Dynamics-aware hybrid offinc-and-online reinforcement learning. Advances in Neural Information Processing Systems,
2022.
Haoyi Niu, Jianming Hu, Guyue Zhou, and Xianyuan Zhan. A comprehensive survey of cross-domain policy transfer for embodied agents. ar preprint arXiv:2402. 200.000000045800 2024.
Haoyi Niu, Tianying Ji, Bingqi Liu, Haocheng Zhao, Xiangyu Zhu, Jianying Zheng, Pengfei Huang, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. H2o+: an improved framework for hybrid offlineaand-online rl with dynamics gaps. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 1421-1428. IEFE, 2025.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in newral information processing systems, 29, 2016.
Art B. Owen. Monte Curlo theory, methods and ezamples. https://artowen. su.. wow.domain//cc////oomm
2013.
Sergey Pankov. Reward-estimation variance elimination sequential decision processes. arXiv preprint 2XXi:1811.06222 2018.
Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic varianco-reducod policy gradient. In International conference on machine learning, pp. 4026-4035. PMLR, 2018.
Matteo Papini, Malteo Pirotta, and Marcello Restelli. Smoothing policies and safe policy gradients. Machine Learning, 111(11):4081-444) 4137, 2022.
Scohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. arXiv preprint urXi::2506. 04168, 2025.
Benjamin Peherstorfer, Karen Willcox, and Max Gunzburger. Survey of multifidelity methods in uncertainty propagation, inference, and optimization. Siam Review, 60(3):550-50) 5910 2018.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Jan Peters and Stefan Schaal. Policy gradient methods for robotics. In 2006 IEEE/RSJ International Conference on Intelligent Robols and Systems, pp. 2219-2223) 2006. doi: 10.1109/ 10.1109/TROS.22 TwT.
Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Policy gradient in lipschitz markov decision processes. Machine Learning, 100(2):255 283, 2015.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Gtable-baselinesh, Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1-8, 2021. URL htttp://mlrr org./papers//22220-1388 html
Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. arXiv preprint arXiv:190600179, 2019.
Herbert Robbins and David Siegmund. A convergence theorem for non negative almost supermartingales and some applications. In Oplimizing methods in statistics, pp. 233 257. Elsevier, 1971.
Andrei A Rusu, Matej Večerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim- to-real robot learning from pixels with progressive nets. In Conference on robot learning, pp. 262-270. PMLR, 2017.
Gilbyun Ryou, GeofTrey Wang, and Sertac Karaman. Multi-fidelity reinforcement learning for time-optimal quadrotor re-planning. Xiv preprint 3rXiu2403.081122 2024.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 1rXiv:1506.02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXio preprint Xiv: 1 107777000444 06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402. 200000000000 2024.
Laura Smith, J. Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In 2022 International Conference Robotics and Automation (ICRA), pp. 1593 1599, 2022. doi: 10.1109/ICRAA460 (tt.:/.....2..988811 2022.9812166.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiu preprint arXim:1507.008: 2015.
Varun Suryan, Nahush Gondhalekar, and Pratap Tokekar. Multi-fidelity reinforcement learning with gaussian processes. arXiv preprint Xivv1112.00488, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. University of Massachusetts Amherst, 1984.
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martin-Martin, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. Annuall Review of Control, Robotics, and Autonomous Systems, 8, 2024.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Matthew E Taylor, Peter Stone, and Yaxin Liu. Transfer learning via inter-task mappings for temporal difference learning. Journal of Machine Learning Research, 8(9), 2007.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23 30. IEEE, 2017.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026 5033, IEEE, 2012. doi: 0T0.0/9/RROS2002..6336
George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning. In International conference on ma- chine learning, pp. 5015-50244 PMLR, 2018.
Linh Le Pham Van, Hung The Tran, and Sunil Gupta. Policy learning for off-dynamics RL with deficient support. In AAMAS, 2024.
Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, pp. 538-584-80 545, 2001. Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-based reinforcement learning. In
RJ Williams. Toward a theory of reinforcenennt-earriing connectionist. systems. Technical Report, 1988.
Ronald Williams. Simple statistical gradient-follownng algorithms for connectionist reinforcement learning. Machine learning, 8:229 256, 1992.
Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mor- datch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized base- lines. In International Conference on Learning Representations, 2018.
Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross- domain policy adaptation via value-guided data filtering. Advances in Neural Information Processing Systems, 36:73395 73421, 2023.
Patrick Yin, Tyler Westenbroek, Simran Bagaria, Kevin Huang, Ching-an an Cheng, Andrey Kobolov, and Abhishek Gupta. Rapidly adapting policies the real world via simulation-guider fine-tuning. arXiv preprinl ar 1rXi:22002.0205,, 2025.
Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost)) locally optimal policies. STAM Journal on Control and Oplimizalion, (8(66)3586-366) 2020.
Tingting Zhao, Hirotaka Hachiya, Gang Niu, and Masashi Sugiyama. Analysis and improvement of policy gradient estimation. Advances in Neural Information Processing Systems, 24, 2011.
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSC1), pp. 737-744. IEEE, 2020.
Yuanyi Zhong, Yuan Zhou, and Jian Peng. Coordinatewwise control variates for deep policy gradients. anXiv preprint 2xXi:2107.04987 2021.
A Complete MFPG REINFORCE Algorithm
Sttorithmpprsenee the complete algorithm block for the proposed multi-fidelit variant of the REINFORCE algorithm.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm 2: Multi-Fidelity Policy Gradients (MFPGs) REINFORCE
Input: Initial policy Te,, value network Ve, multi-fidelity environments M and M batch size for correlated trajectory samples N batch size for control variate sample mean estimate N step sizes {ax}, maximum number of iterations N.
Output: Trained policy T
for h 1==1a to N do
Current policy at iteration k: Tez (a 8)
Collect N correlated Irajectory samples Di {-. +/J using To.- // See Algorithn
Collect N additional uncorrelated low-fidelity trajectory samples D2 1r/ using mea
Compute correlated policy gradient random variables using D
T-1 G.
vo((tt)) log To, (a s
T
x
Σ (G - Vo(s),)) log To (a st.).
Estimate the optimal control variate coefficient & using D1 and Eq.
Estimate the control variate sample mean using D2: A = N Σ j== p'
N'
Construct MFPG estimator: Ve, + MFPG
gh
Na Vo
(X 76t -Ph)].
Perform policy update 0k+1 + 02 + 32 MFPG
10
Fit the value function network Vo using high-fidelity samples in D
11 return policy Wax
B Proof of Lemma 1
The proof of Lemma follows standard control variate arguments (Owen 2013 Ch.8) and proceeds follows.
Proof. We first establish unbiasedness of the control variate estimator for arbitrary coefficient. IR. Taking expectation for the defined multi-fidelity control variate random variable over the joint probability space Ω defined in Section yiclds
Eunn (7 (c)] = Eu (X X c(X - n²)
= Ew + c(Eunn [X v
=0
Eu-osXXi
where sampling from the joint probability space ~ Ω reduces to sampling single-fidelity trajectories 1h M (TT) and ~ MM(TT)) when evaluating the expectation of the corresponding single-fidelity random variables X and X respectively.
Next, we examine the variance the multi-fidelity control variate estimator:
Var (c)) = Var(X,t + o(x; - p²))
Var(X + cx,²²)
Var(X + Var(X + (C,,
26

---

Published in Transactions on Machine Learning Research (02/2026)
where the second line uses the fact that addin//subtractinn constant does not change the variance of random variable, and the third line uses the standard formula for the variance of linear combination of random variables. The variance Var( (ar(2*(cc) defines a quadratic function in e, whose minimizer can be found by sctting
Th ,h, x11) Cov(X
Var(X
V. Var (c)) (c)) =0 c'
p(x 78 Xpe
Var(X??)
At the minimizer c², the variance of the multi-fidelity control variale estimator is
Cov to 7hi X
2Cov2 x
Var (c)) Var(X
Var (x71)
Var(X7)
( R0 X 2 Var(X Var(X
Var(XX))
Var(X,")
(1-p² (1-P2(XTR) Var()))
C Proof of Theorem 1
The proof of Theorem leverages the Tipschitz continuity of the policy gradient for policy deployed in the MDP M under assumptions Assumptions I to Under these assumptions, the Lipschitzness of the policy gradient was proved for the infinite horizon setting by Zhanng al. (2020). We follow similar procedure to establish the Lipschitzness for the finite-horizon case, stated below.
Lemma 2 (Adapted from Zhang et al. 2020)). Under Assumptions 1 to 4 for a problem of horizon T, the high-fidelity only policy gradient (() ET~Ma(0) [R(T)] Li-sschchitz, where
T+1 + (T + 1) UhLo 1 - +T++ 2)
LT
(1-))²
(T 2) (T + (T + 1)7 T+2) (1 7) 27 (1 (7
(T 22y + +T+
U...
(1 71
Proof. Note that the high-fidelity only policy gradient can be written as
T T
JJ(0)
ΣΣ
Σ / nd (olinons ditx) Vo log
bt
po.0
dobarx dobicun
1= ==0
Ltx
where po,0t+*x
p 16 dni))
II
27

---

Published in Transactions on Machine Learning Research (02/2026)
Thus, for 02 E RR, we have for the high-fidelity only policy gradient,
T-t
/ (1"(oita) V log no (al 18/))
[VJ( - VJ(02)|1
V log Tay Tag(ah | pē, 0.04
1= K=0
A
1hh (Bftic) x / log To, (a stt) (pb, aatt - per d dabete
c+x (6btecr abtw! Voo no, (at (0)) V log Ta (ay ( Por out tnneendooenn de
1m km0
/
( aftaal
Péz 0:t40 do dapet log To, (a st)] Ipb 0eexx
(5)
From Assumption 4
we have
||| s L - 82
L || 8²
(6)
Further, note that
[+
pl 1 att) )
++ II 2
L+
h Po. 0xt+x
h 0.10
II k==0
Tox
(a (
II N
To= (a (o))
From Taylor's theorem, At E (0, 1) such that for 1 = 20 + (1 2)0?, we have
14x
II
nox (a
(0' 02)
Σ
VOTT ((<()())))
II m=Ouuhh
Ta(at((()
S 110 0
Vo log (<)))>>))
II <a(a?)s
II ++
$ (1+*+11 Bg BB.1 0al
(
(Assumption 3
12 s (1+*+1)-UU Uk BBg 100 1
(7)
From Equations 55 to
7
we get
T T
ΣΣ
t+x +
+U + 1nttt) 110 021
1=
Using the fact that
T T-E
t+ 1- (T
1- (T 2)- (T + 1),T+2
(1 71
(1- (T' + 2- (3+ 1)2) (1-7)
ΣΣ (1+++11 0=0
(1 11-93
27 (1 (T + 2)
T+1 + (T + 142
(1 7)3
28

---

Published in Transactions on Machine Learning Research (02/2026)
get that
LT 1 (T +2+ + (T + T URLO
71-
(1- 2) + (T (T
T+1 + 1)7 (1 7) + 2y (1 (T + 2 + +T 147)
U R
(1 (1-
Note that in the limit of infinite time horizon, recover the Lipschitz bounds proposed by Zhang el al. 2020) for the infinite horizon setting, given by
lim LT 1 L + (1++) T-000
B.
(1 71-
(1 (1-
To establish the asymptotic almost sure convergence of J[ MPPG)) I|| to 0, we also need the following preliminary result, which is an adaptation of a similar result from Zhang et al. (2020). but for the multi- fidelity case.
Lemma 3. Under Assumplions to 4 the norm of the Multi- Culti-Fidelily Policy Gradient estimator is bounded, i..., 3 2 0 such that || JMMPGG (0) < a E Rdd Further, Consider the random variable Wk, defined as
W& = J(OMFGG) LT
2
aj.
j=k
Let Fy be filtration denoting all randomness in k iterations. Then, we have that
E 2 We az 0 (.
Proof. We first show the boundedness of the MFPG estimator. We have
Uh T
Uh
1 1-7T
T
T Gh V log Togmo(al (a (.)
T
T
1 1-7
1=0
Ul Be
T-1
UlBa
1 - 7p
VX 00
G V log To (af,e s/,
7
T
1 7
0
N
V ト!,
- - VX D1
U
1 1-
+ 2e
Ul Be
1- 1-
= 0
4h
TNh
1-
TNh
1-9
29

---

Published in Transactions on Machine Learning Research (02/2026)
We now show the stochastic ascent property associated with Wk. By Taylor's theorem we have
W J(OMFFG)) (0N4PPG @MPPG) VJ(@s MPPG)
L Σ
aj
1=k+1
= =](OMERG) + (0NfTe M VJ((N
0c + CONIERG 0g / (VJ(O, MPPG) (J(O)FPG)) - I.T 22
2 J(OMEEG
+ (offree PFFG T (J(OMP)G)- LT 10
0g P
- LTl2 Σ
j=++1
E[ FR] 2 J(0
+ 0
1NFPPG
0%
FR
- L, |IOMEPG
0
MF
[F
Ly
k+1
=J(OMEPG E 1 0
M
F
VJ((O
x Σ
LT (O L
j=
2 J(OMEFG + E [0 0 Fi]
VJOOPPPG)
- Lrafe= L Σ og
j=k+1
= BJOOOPP 2 J(OMFPG) + E [0 [MMPPG 0
MPG
FR
(JOMPPG
TTpa
To show the bound, we have E - OMFGG FR] E JMFPS This is the crucial step dependent on unbiasedness of control variates, which allows us to conduct this analysis. Putting this in the above incquality gives us the desired bound.
oa JOMMPP))
Using Lemmas 2 and we can now formally prove Theorem [ which we restate for the reader's convenience.
Theorem 1 Under Assumptions 1 to let (0MEPG) JeEz,, (0)) koz] be the sequence of policy parameters generated by running the REINFORCE algorithm with MFPG and high-fidelity only policy gradient estima- tors. Then, for problem with time horizon T, after N iterations, with step sizes for both iterates taken to
be a sequence ag satisfying Σ a = co, Ex ax < 0c, we have
J(0") J(01) + (1-p2) p2) dZL N 2k a
min E MONOGE KEIN
ΣN Okk
where Lr is the Lipschitz constant of the high-fidelity policy gradient, established in Lemma g Moreover, we recover the corresponding rale for the high-fidelily only policy ilerales JKEEt by substibuting p 0. Finally, we have that limg E ||| [
= =0 almost surely.
Proof. Let the MPPG estimator be written (MFPG = VJ(0) + E. and the high fidelity only policy gradient estimator be written as (0) == V.( We analyze the sequences (QMFPG) ) Jazz and {0) hezi generated by the two estimators respectively,
MFPG MMPG aak MFPG k Z+.
0 = 0% + +ag Fh, 0- OMPPG :-
30

---

Published in Transactions on Machine Learning Research (02/2026)
LT
J(0k+1) 2 ((0%) + (VJ(O))) 02-1 022
10% 0211
2
- -((()) + au 13) a
|1Ai1112
2
a3.L
-(O + au (V) v)(O2) + Ex)
[V + Exl??
2
IT |E
-(()) + a4
|b + (ax -agLT) (VJ Ex) 2
2
ofLT
[ s 1(0+11) - ((0)) (ax= - Lb (VJ(0)), Ex) + a 11Ex/11 1
0k
afLlT)
N
|| S 3(22+1) - -(())
Σ
(az - o (VJ(O)))
Okk 2
k=1
k=1
Lo
11Ex11
2
km
LT 11 112
1(00)-J(8)) Σ
(V Ex) +
2
Similarly, for the MFPG itcrates, we get
ak aBLTT)
N
1(0)) (10)--(01))
Σ
- -ax-a2
(VJ(OMFPG) Ex) +
a LT || 112
2
By taking expectation over the randomness for both sequences, from Assumptions and and the fact that the MFPG is an unbiased estimator for true policy gradient, we get
N
agL 2
S ((00) j(0i) + 0² La
E [IVJ
2
ak
0
5 1(00) ((0i) + LLLr
min [[
2k=1 0K
Σx=1 ak
a bT
N
LT 2
LT
N
[ [OM < J(0') - J(0i) + (1 - --²
0k
2
JJ MFPM 117 < J(0') J(Oi) + (1 2²) N
a² Lo
N k=
0
min KE
a L1
a2
We establish the global asymptotic convergence of the multi-fidelity policy gradient using the arguments developed in Zhang al. (2020) To show that limę- [V = 0 almost surely, we first establish that lim inf VJ = 0. Let J' be the global optimum, we have that W₂ S J by definition of
We. From the stochastic ascent property and corresponding bound established in Lemma 3 we have
E - WK+1 [F) S (J" - W - au (OM
0k VJPMPP y| < almost surely.
(supermartingale convergence theorem (Robbins & Siegmund 1971 limiif (0 = 0.
as J W, 2 0.) (: >i a o0)
k
31

---

Published in Transactions on Machine Learning Research (02/2026)
Finally, we will establish by contradiction, that sup--ae [ O. Assume that 3 c 0 such that lim - [| 6. Thus the set N1 [0/PPG [VJ 2 2//3 is an infinite set. Because lim on [| O, the set N2- = {OM |VJ I 4/3} is also an infinite set. We can define the distance between the two sets D(N1,N2) = infø, EN, info,eN, ||0 02l1. which by assumption will be positive. Because both N1, N₂ have an infinite number of sequence members, there must be an index set such that the sequence (0 (#MPPG) jkez switches between N₁ and N₂ infinite times, and there must be indices {si) {4:h20 such that 0s, E NN, 2 0, 0,0,, E N2, i 2 0, and the iterates (MMPPO) Jkez Jeek (OMFPG ....
....0 0 are neither in N1 nor in N2. Thus, have
4
Σ D(Ni, = Σ MPMP 8MFPGI = Σ 20
Σ
MMPPG - @MFPG) (0
10
10MPGG @MFPGI
120 K=
(8)
REI
We know that
S 1VJ(0L))
Og. S Σ
[aIIVJOOMMPG)) <0
9
RET
KET
< 00
KEI
Σ E [[O
Σ a [V
<0
(because E V < 2
KEI
KEI
J0MEPG
@ < xo almost surely,
(monotone convergence theorem)
KEI
which leads us to a contradiction in light of Equation g and thus lim s [J[(M ]| = =0 almost surely. Thus, we get that 1im [ (FG))) =0.
Additional Ablation Study Results
This section presents additional ablation results that complement Section 6.5
Reparamcterization trick. We examine the effect of the reparameterization trick described in Section on our MFPG method. Figure I0 shows evaluation return curves for MFPG with and without the reparame- terization trick, along with the High-Fidelity Only baseline, on Hopper-v3 with 0.5x and 1.2x friction shifts. When the dynamics gap is mild (cf., the 1.2x friction setting in Hopper, right panel), reparameterization substantially accelerates Icarning-for example, around 0.2M steps, the confidence intervals of MFPG with reparamelerization lie well above those of its counterpart without reparaneelerization. In contrast, when the dynamics gap is larger (cf. the 0.5x friction setting, left panel), the difference much less pronounced.
This pattern is intuitive: when low-fidelity dynamics are close to those of the high-fidelity environment, stochasticity in action selection dominates; reparameterization controls this source of randomness and pro- duces highly correlated rollouts. In the extreme case where the low-fideliyy dynamics are perfect (and deterministic), reparameterizztion makes the low- and high-fidelity rollouts identical, so that X and X² cancel out. The policy gradient estimate then reduces to y², which is estimated with abundant (perfect)
32

---

Published in Transactions on Machine Learning Research (02/2026)
low-fidelity samples. By contrast, when the dynamics gap is large, discrepancies in transition dynamics dominate, and even with reparameterivation, divergence between high- and low-fidelity rollouts persists, resulting in weaker correlation.
MEPC Inoibeed reomstsiiiig
MEEG twish recemstniiiing
Iligh-iiallt Only
MIPEC twrtheet meonetsmiingg
MPFG (with
Fridlity
690
Imparsmeeettation) MFFC reparectetiaaiionn Highh-idellly Only
Iso eeponindettiaaiieo
Ieparameettsatiio)
100
Kosund
640
300
710
HiighPidelity Only
400
600
600
540
204-
-000
600
300
High-Fiddiit Envirenment Stage 109
Hligh-lidellyy Enstovement Bogger Elictios
200
Happer (.5 tittien
100
02
3.5
056
0.2
2.4
06
Figure 11: Evaluation return curves for MFPG with and without periodic roconstraining of low-fidelity trajectories versus High-Fidelin Only on Hopper-v3 with 0.5x (Ieft) and 1.2x (right) friction. Solid lines indicate bootstrap point estimates (means), and shaded regions denote two-sided 95% bootstrap confidence in- tervals. Reconstraining can accelerate learning in some settings for example, around 0.05M steps in the right panel- but can also harm performance in others, e.g., during the second half of training in the left panel. Accordingly, we do not employ this trick in our evaluations of MFPG.
High-iddelt Enviiommnnt Stops Ses
Hidhllyy Envirnment Steps. Tee
Dapper Eticion
Hupper fridion
Figure 10: Evaluation return curves for MFPG with and without the reparameterization trick versus High-Fidelity Only on Hopper-v3 with 0.5x (left) and 1.2x (right) friction. Solid lines indicate bootstrap point estimates (means), and shaded regions denote two-sided 95% boot- strap confidence intervals. The reparameteri- zation trick proves especially critical in mild- gap settings (e.g., 1.2x friction).
Varianco-bias tradc-off. In practice, we observed that periodically reconstraining (the correlated) low- fidelity rollouts back to their high-fidelity counterparts can strengthen correlation and accelerate learning in some settings (i.e., every few steps along a low-fidelity rollout, reset the low-fidelity state to match the state of the correlated high-fidelity rollout at the same time step). For example, in the 1.2x friction case in Fig. 11 (right panel), around 0.05M steps, the MFPG variant with periodic reconstraining attains an evaluation return whose confidence interval lies well above that of nominal MFPG without reconstraining, However, this procedure may also introduce bias into the policy gradient estimator, leading to degraded performance in certain cases (c.g., the 0.5x friction setting in Fig. 11 left panel)-the confidence interval of MFPG with reconstraining mostly lies well below that of nominal MFPG without reconstraining throughout the socond half of training- These findings highlight practical variance- trade-off. We do not adopt this reconstraining trick in our experiments. Developing systematic, unbiased mechanisms for drawing even more highly correlated samples across environments remains important direction for future work.
E
Additional Experimental Results on Off-Dynamics Tasks
For the 15 main task settings in Section 6.3 evaluated with 20 random seeds per method, Fig. 12 reports the mean difference in AUC of each method relative to the High-Fidelity Only baseline (so 0 corresponds to High-Fidelity Only), computed analogously to Fig. Bars show the bootstrap point estimate of the mcan difference, and error bars denote two-sided 95% bootstrap confidence intervals. As a complement to Fig. 5
Fig. 13 reports the mean difference in evaluation return between MFPG and the High-Fidelity Only baselin..
Figures 14 and 15 further show box plots of the final evaluation return and AUC for each method and setting across the 20 runs: the center line indicates the median, the box spans the interquartile range (25th-75th percentiles), and whiskers extend to the 5th and 95th percentiles; outliers beyond the whiskers are not shown.
33

---

Published in Transactions on Machine Learning Research (02/2026)
Off-Dynamics Task Comparisons: AUC (Relative to High-Fidelity Only)
DARC
Only (10b))
1.5
0.0
0.5x
0.8%
12x Hopper-gravity
2.0%
5.0k
AUC
5.0
0.0
0.5x
0.8%
1.x
2.0x
5.0x
Walker2d-frittio
AVC
0.3x
0.8%
1.2x HalfCheetah friction
2.0%
5.0x
Figure 12: Mean difference in AUC of each method relative to the High-Fidelity Only baseline (0 corresponds to High-Fidelity Only) across the 15 main task settings in Section 6.3 computed analogously to Fig. 3 Bars show the bootstrap point estimate (mean), and error bars denote two-sided 95% bootstrap confidence intervals.
Figures 16 and 17 further reports results on 24 additional (ask settings, each evaluated with 5 random seeds per method. Due to the small number seeds, these additional results are primarily used to illustrate the consistency of qualitative trends (c.g., the prevalence of performance collapses), and we do not draw formal statistical conclusions from them. Instead of reporting full distributions, in these figures, bar heights indicate medians and error bars denote the minimum and maximum values across seeds.
In Figs. 16 and 17 kinematic variations are categorized as casy, medium, or hard; each setting constrains a specific joint. to reduced range of motion to mimic actuator damage. For kinematic shifts, we consider cases where the agent's thigh joint is impaired (leg joint for Hopper-v3, since the original benchmark does not include a thigh joint shift for Hopper).
Finally, in Figs. 14 and 16 More High-Fidelity Data (15x) is an reference baseline that trains with access to 15x more high-fidelity samples. Dashed lines denote the median performance of More High-Fiieeity, Data (15x).
In Fig. we observe that, in general, High-Fidelity Only struggles to match the performance of the More High-Fidelity Data (15x) baseline, highlighting the clear benefit training on larger high-fidelity sample sizes. Nevertheless, in some settings Low-Fidelyty Only (100x) can match or even exceed the More High- Fidelity Data (15x) baseline, albeit often with substantially large cross-sced variance (e.g., Hopper Hopper-gravity 1.2x). This does contradict the role of the More High-Fidelity Data (15x) baseline as a strong reference point: in certain regimes, the dynamics mismatch (bias) can be benign, and abundant (100x) low-fidelity interaction can provide learning signals that transfer unusually well to the high-fidelity environment. In such cases, aggressively exploiting the low-fidelity data can produce large gains, but this behavior is brittle-wie when low-fidelity data become harmful (cf. HalfCheetah), these approaches can degrade substantially. Moreover, the benefit of low-fidelity data is difficult to predict a priori (e.g., Low-Fidelity Only does not vary mono- tonically with the dynamics gap). In contrast, MFPG provides a more consistent and reliable mechanism for leveraging low-fidelity data by anchoring updates to high-fidelity gradients while using low-fidelity data only variance reduction tool.
gravity
34

---

Published in Transactions on Machine Learning Research (02/2026)
230
200
20
1.30
Reward
100
50
-30
-00
0.0
0.2
0.8
0.8
1.0
0.0
0.2
0.8
0.6
0.5
0.4
Eligh-Pidelity Environment Steps
Eldgh-Pideltty Enstroument Steps 1e6
High-Fidellty Envirnment Steps
dynamics shift
smaller (0.8x)
larger (5x)
Figure 13: Mean difference in evaluation return of MFPG relative the High-Fidelity Only baseline (0 indicates no difference) for the settings in Fig. Solid lines show the bootstrap mean estimate, and shaded regions denote two-sided 95% bootstrap confidence intervals.
Off-Dynamics Task Comparisons Final Evaluation Return
High Fideliy Only
MEPG
DARC
PAR
Low Low-Fidelity Only 1100
--- More Fidelly Data (15k)
Peeformance Collapse Threshold
Final retur
500
250
Hopper gravper-gravity
Final Final retur
600
0.5%
0.8x
1.2v Walker2d-rrittinn
2.0x
5.0k
Final return
2500
0.5x
0.8x
1.2x HalfCheetah-frrition
2.0x
5.0x
Figure 14: Box plots of the final evaluation return for each method across the 15 main task settings in Sec- tion 6.3 The center line indicates the median, the box spans the interquartile range (25th 75th percentiles), and whiskers extend to the 5th and 95th percentiles; outliers beyond the whiskers not shown.
Off-Dynamics Task Comparisons
AUC
High Fidelity Only
MFPG
DARC
PAR
Low. Eddcity Only (100)
Pesfomanc: Collapse Threshold
AUC
0.8x
1.2x Hopper-gavvtt
2.0x
5.0x
1c8
AUC
6
1c9
Walker2d friction
AUC
0.0
HaliChestah-ffiit friction
Figure 15: Box plots of the AUC for each method across the 15 main task settings in Section 6.3 The center line indicates the median, the box spans the interquartile range (25th 75th percentiles), and whiskers extend to the 5th and 95th percentiles; outlicrs beyond the whiskers are not shown.
35

---

Published in Transactions on Machine Learning Research (02/2026)
Off-Dynamics Task Comparisons Final Evaluation Return
HighFidelly Fidelly Only
MFPG
DARC
PAR
Lowe Fidellyy Only (1000
--e More High Fidelity Data
Perfremance Collayse Threshold
Final retum
800
2.0x
5.0x
easy
medium Hlopeer-klneman
hard
Pripper-triction
Final return
1600
800
0.5%
0.8%
12x Walker2dd gravit
2.0x
5.0x
medium
hard
Walkerld kinematteeee
Final return
2000
easy
Hal/Cheelah-gravity
HallCheelab--Ktnemalle
Figure 16: Final evaluation return of each method across additional 24 task settings. Bar heights indicate medians, and error bars denote the minimum and maximum values across seeds.
Off-Dynamics Task Comparisons AUC
1e8
Hidbliidelty Only
Low- Red-iilllty Only
Performance Callapar
AUC
0.5k
0.8k
1.2x
2.0%
5.0x
medium Hopper- Kinpper-Kinemaicc
hard
1e0
Hopper-fricte friction
AUC
1.0
0.5
0.0
0.5k
0.8k
1.2x
2.0x
5.0x
medium
hard
Walker2d-gravlty
Walker2d -iinmatic
12
AUC
ddd
0.5x
0.8k
HalfChettah-graviig
1.2g
2.0x
5.0x
easy
medium HallCheetah CinCheetth-kimemattee
hard
Figure 17: AUC of each method across an additional 24 task settings. Bar heights indicate medians, and error bars denote the minimum and maximum values across scods.
36

---

Published in Transactions on Machine Learning Research (02/2026)
F Implementation Details and Hyperparameter Setup
This section provides implementation details and the hyperparameter sctup for all evaluated approaches. Details of the ofl-dynamics task settings described in Section 6.1 are available in the ODRL benchmark (Lyu et al. 2024b). Table summarizes the hyperparameter configurations for all evaluated methods. To ensure reproducibility, we include our code in the supplementary materials and will release the project as open source upon acceptance.
We build our implementation on top of the RL library Stable- 3aseliens (Rafiin et al. 2021 For exper- imentation, we use random seeds {0, 1, 2} for hyperparameter tuning and {3, .... .....
22 for evaluation in Sections 6.3 to 6.5 (f3, 4, ... 7 for the additional 24 tasks in Section E The High-Fidelity Only baseline is tuned using 3 seeds {0, 1, 2). The resulting configuration serves as the shared backbone for the other methods, following standard practice in the off-dynamics RL literature (Xu et al. 2023 Lyu et al. 2024a). All additional hyperparameters are tuned with 2 seeds f0, 1}, either over more task scttings (for the multi-fidelity methods) or longer training steps (for More High-Fidelity Data (15x)). All single-fdellity methods are tuned on the MuJoCo Hopper-v3, Walker2d-v3, and HalfChoetah-v3 tasks with original dynam- ies, chosen because they represent the midpoint among the varied dynamics settings. Multi-fidelity methods are tuned on the same three tasks under a representative subset of dynamics variations, namely 0.8x and 2.0x gravity. We verified that the tuned hyperparameters generalize well to other variation types and levels (cf. Section E For the extra sensitive hyperparameters of PAR (reward-angmennttiio weight) and DARC (the standard deviation of Gaussian noise added to classifier inputs), we additionally lune them on the 0.8x and 2.0x friction variations of the three tasks. We note that this protocol grants the baselines more tuning privilege than MFPG, highlighting the simplicity and minimum tuning overhcad of MFPG.
High-Fidelity Only. We implement the standard REINFORCE algorithm Williams 1992 with state-
value subtraction. Subject to the high-fidelity sample restrictions per policy update described in Section 6.1 we perform grid search over the learning rate (1 11x10-4, 2x 10-4, 501001 7x 10-1, 10-1,90 90110-,, 1 1e11xx00x0 10-3, 2 10 3), discount factor y {0.95, 0.97, 0.98, 0.99, 0.995), advantage normalization (True, False), maximum gradient norm {0.5, 1, 2); and stato-value loss weight (0.25, 0.5, 1}. We adopt the default network architectures for the policy and state-value networks in configuration is then adopted as the shared algorithmic backbone for all multi-fidelity methods. The full hyperparameter configuration, along with those for all other methods, is provided in Table D
(Ralfin al.) (021). The tuned REINFORCE
DARC. DARC Eysenbach et al.) trains two domain classifiers 48aas (high St, as st+1) and goaa (high I St, 0z) to predict how likely an observed transition (8t, 0t, 8141) or state-action pair (St, az) comes from the high-fidelity environment. The classifiers are trained by minimizing cross-entropy losses over the replay buffers Duigh and Diow of high- and low-fidelity samples, respectively:
LSAS(OSAS) = -E [log 4osa (low 8t, ae, 11++ - EDa [log yaass (high 8t, Ues 8141)] (OSA)))) = EED... [log 4 (low St, at) - EDaigh [log 48ga (high
| St, az)].
(9)
The logits of the trained classifiers are then combined to approximate the dynamics gap, log p(éralle high) for each observed (low-fidelity) transition (St, at $1+1):
p(ec 1 low)"
Ar(s; at, 8++1) = [high 8t, az 8++1) poig(high| | st, log p(oo ae, 80+1) + log | 88, at). (10)
This estimated dynamics gap is used to augment low-fidelity rewards r raugment, (8/, at 2(41) = r/(ss, af 1(41)+ +
Ar(s) af s442). The augmented rewards penalize the agent for exploiting low-fidelity transitions that are implausible in the high-fidelity environment.
We adopt the standard implementation from the ODRL benchmark cross-validated against the authors' original
com http:///ihbbbbomm//oymaaal////// code (https:/ /ith/b.. com/ google-researeeeee / rowggle-eseearc/tteeeeee basssarcch/rree/mmttr//arrr on-policy REINFORCE algorithm, we retain the original classifierlearning and reward-angmentation mechanisms, substituting only the policy optimization backbone. At each update, classifiers are trained using off-policy samples from the high- and low-fidelity replay buffers, as in the original implementation,
To adapt DARC to the
37

---

Published in Transactions on Machine Learning Research (02/2026)
by minimizing the cross-entropy losses in Eq. (9) The learned classifiers are then used to augment the rewards, ef., Eq. (10), for low-fidelityy, on-policy samples, which are subsequently employed to compute the REINFORCE updates and learn the state-value function.
Building on the shared REINFORCE backbone used by the High-Fidelity Only baseline, we further tune several DARC-specific parameters which the method is more sensitive: the standard deviation of Gaussian noise added to classifier inputs a regularizer (an effective stabilization trick noted by the authors (Ey- senbach et al.b)) {0.5, 1, 2), and the number of warm-start, high-fidelity environment steps {2000, 10000, 50000} prior to classifier learning and reward augmentation. For other syperparameters--suee such classifier architecture, learning rate, and batch size-we adopt the default values from the original paper (Eysenbach| et al.). The resulting classifier training curves are stable.
PAR. Similar in spirit DARC, PAR (Lyu et al. 2024a also estimates the dynamics gap of low-fidelity transitions for reward augmentation. However, instead of domain classification, PAR adopts representation- learning approach: it trains state encoder 21.t fs(ss) and state action encoder 22, gr(21, at) by minimizing a latent dynamics consistency loss over the high-fidelily replay buller:
L(C,V) = (s of ( [I(e)(((()) at) (
(11)
which essentially learns a dynamics model for the high-fidelity environment in latent space. The trained encoders are then used to estimate dynamnic mismatch for low-fidelity transitions and augment the rewards:
A Faugment, (8/s o2 s/41) = r/sh de s(41) - (Is - - fc((t+1)113
(12)
where 3 is weighting hyperparameter. Furthermore, unlike DARC, which trains its pol- iey and critics only low-fidelity samples with augmented rewards, PAR trains its policy and critics on both high- and low-fidelity samples, with reward augmentation applied to the latter. We adopt the standard implementation from the ODRL benchmark (https://git http:///tttuuhhooom www..ooo///ooiiiieie.....m OffDyyamiisRR.//fff dynamics-rl)), cross-validated against the authors' original Analogous
PAR Bettmm00)
PAll (beteod.5)
PAR (hatart)
PAll (hets-2)
Final Return
code (https:/ / githtttthh..oo.. com com//oom//mceeeeeeee /dmkjjt/PAAR our adaptation of DARC to the on-policy REINFORCE algorithm, we adapt PAR by retaining its original mech-
1200
600
representaaioon-learingg and reward-augmentatiin anisms while substituting only the policy optimization backbone. At each update, we draw off-policy, high- fidelity samples from the replay buffer to train the encoders by minimizing Eq. [1].. The trained encoders are then applied to augment rewards for on-policy low-fidelity samples, in Eq. (12 Unlike DARC, which relies solely augmented low-fidelity samples for policy and value function learning, PAR leverages both on-policy high-fidelity and augmented low-fidelity samples for REINFORCE policy updates and state-value function learning, consistent with the original PAR algorithm.
Walker2d 2.0x friction
HalfCheetah 2.0x friction
1c9
0.5
0.0
Walker2d friction
HalfCheetah 2.0x
Figure 18: Sweep of the reward-auugnentation weight 3 for PAR on two tasks. The relative per- formance across 3 values varies substantially be- tween tasks. Bar heights indicate medians, and error bars denote the minimum and maximum values.
On top of the shared REINFORCE backbone, we further tune the reward-augment angred-augmentttion weight / in Eq. 112 over
the grid (0.1, 0.3, 1, 2), as PAR is particularly sensitive to this parameter; cf. Fig. 18 The original PAR (Lyu et al. 2024a) employs a per-lask weight for this reason, and we follow the same convention, reporting the We note, however, that this requirement introduces a computationally expensive adaptation overhcad, unlike other algorithms, which do not require per-task tuning. For other hyperparameters- such as encoder architecture, learning rate, and batch size- we adopt the defaults from the original paper [Lyu et al. 2024a). The resulting encoder training curves stable.
selected 3 for each task in Table [
38

---

Published in Transactions on Machine Learning Research (02/2026)
MFPG. Building the shared REINFORCE backbone, we perform grid search over the exponen- tial moving-average weight 7ma in Fq. 3 using values {0.92, 0.95, 0.99). We find that MFPG's median performance consistently exceeds that of the High-Fidelity Only baseline across the tested values of Maai even though the absolute performance metrics vary with Mma In off-dynamics task scttings, we drop the control variale term cf cx="" (+) whenever the estimated correlation coefficient for the current batch, i.e., p (X , x in Eq. ( is negative. Since high- and low-fidelity samples should in principle be posi- tively correlated in off- dynamiss settings, negative correlation suggests either low-quality samples or noise in correlation estimation. In practice, such cases occur only rarely. This engineering trick helps mitigate noise in estimating the control variate coefficient, which can arise from extreme low-fidelity samples. Developing more principled estimation schemes remains an orthogonal direction for future work.
Low-Fidelity Only (100x). The Low-Fidelity Only bascline serves ablation of high-fidelity samples from the multi-fidelity approaches. Accordingly, we adopt the same hyperparameter configuration as the shared REINFORCE backbone, except that each policy and state-value function update is performed using abundant low-fidelity data in place of limited high-fidelity samples.
More High-Fidelity Data (15x). To ensure fair benefit from the additional high-fidelity samples, we tune the sensitive hyperparameters for the oracle bascline More High-Fidelity Data (15x) separately from the shared REINFORCE backbone. Specifically, we tune the learning rale over the grid {5 10-1,7 x 10 10-4,9 4,9 x 10 4, x 10-3 3,2 x 10 7,4x 10 10-3} and the batch size (i.e., samples per policy update) over the grid {1024, 2048, 3072, 4096, 61443. Because of the expensive, longer training time for this baseline, we first run all configurations for 3 million steps, then select the best 7 configurations and extend them to 15 million steps to determine the final configuration Other hyperparameters are kept consistent with the shared REINFORCE backbone, as they are relatively insensitive.
Low-fidelity data amount. In baseline comparisons, we assume that low-fidelity samples are cheap to generate, and all multi-fidelity methods may supplement limited high-fidelity data with up to 100x additional low-fidelity samples per policy update. While PAR has been empirically observed to benefit from more low- fidelity data in some task settings (Lyu et al. 2024a), we also found it to be sensitive to the amount of low-fidelity data in certain cases. To maximize baseline performance, we additionally tune the ratio of low- to high-fidelity samples per policy update over the grid f10x 210x,20x,10 For DARC, performance is generally maximized with 100x low-fidelity samples. For PAR, performance is maximized across tasks at 20x. The Low-Fidelity Only baseline is assigned 100x low-fidelity samples to match the budget. We implement parallelized low-fidelity environments, each generating 10x low-fidelity samples per policy update; hence, DARC and Low-Fidelity Only employ 10 environments, while PAR uses 2 environments. For MFPG, we did not tune the amount of low-fidelity samples, since MFPG is in theory stable with respect to low-fidelity sample amounts, so long as the low-fidelity sample amount is sufficient to estimate the low-fidelity sample mcan pt. Our empirical results in Fig. 3 support this point. Therefore, instead of using 10 parallel low-fidelity environments (which would exceed the 100x budget, given that MFPG also requires 1x correlated low-fidelity samples), we adopt 9 parallel environments in all main results, corresponding to 90x low-fidelity samples (plus 1x correlated low-fidelity samples). In some auxiliary variance and sensitivity analysis in Sections 6.2 and 6.5 use 100x (uncorrelated) low-fidelity samples; we specify the low-fidelity sample amount separately in those sections. Finally, we emphasize that DARC and PAR receive more tuning budget than MFPG, underscoring the simplicity and low adaptation overhcad of MFPG.
100x) } for PAR and DARC.
Complete hyperparameter configuration. Table ! reports the full hyperparameter configuration for all mothods the bascline comparison results. As noted above, the High-Fidelity Only configuration serves as the shared backbonn for other methods, with algerithm-specifi hyperparameters listed on top of this backbone. Brown entries indicate hyperparameters that replace the corresponding values in the backbone.
39

---

Published in Transactions on Machine Learning Research (02/2026)
Table Hyperparameter configurations for evaluated methods.
High-Fidelity Only (REINFORCE backbone)
Optimizer
Adam Kingma 2014
Learning rate
7 10
High-fidelity batch size
100
Discount factor
0.97
State-value loss weight coef Maximum gradient norm
1.0
1.0
Policy network
(64, 64)
State value network
(64, 64)
Nonlinearity
Tanh
DARC
Classifier network
(256, 256)
Classifier nonlincarity
ReLU
Classifier Gaussian input noise std.
0.5
Classifier optimizer
Adam
Classifier learning rate
10-4
Classifier batch size Replay buffer size (classifier) Low-fidelity data amount / policy update
128
106
100x (relative to high-fidelity data)
Warm-up high-fidelity steps
2000
PAR
Encoder network
(256, 256)
Encoder nonlinearity
ELU
Representation dimension
256
Encoder optimizer
Adam
Encoder lcarning rate
10-4
Encoder batch size
128
Replay buffer size (encoder)
1 106
Polyak averaging cocfficient for encoder updates
0.995 (for the target encoder network)
Low-fidelity data amount policy update
20x
Reward angmentation weight 3
0.5 (Hopper), 0.1 (Walker2d), 1.0 (HallCheetah)
MFPG
Low-fidelity data amount / policy update Exponential moving-average weight 7mna
90 uncorrolated + 1x correlated
0.95
Low-Fidelity Only (100x)
Low-fideity data amount / policy update
100x (no high-fidelity data)
More High-Fidelity Data (15x)
Learning rate
10-4
High- fich-fiielity batch size
3072
40

---

Published in Transactions on Machine Learning Research (02/2026)
G Extending MFPG to Broader Algorithms and Settings
For the first futuro-work direction discussed in Section 7 we highlight two subdirections. First, natural next step is to extend the proposed multi-fidelity control variate approach to a broader class of modern RL algorithms.
A concrete starting point in this direction is modern on-policy actor-critic methods, such as proximal policy optimization (PPO; Schulman el al. 2017). As discussed in Section 4 the MFPG framework can, in princi- ple, be instantiated with other on-policy policy gradient methods by changing how the random variable X is computed from sampled trajectories. In this work, we focus on the classic REINFORCE algorithm to sim- plify the theoretical analysis and establish convergence guarantees. Modern actor-critic methods, however, deliberately trade bias for lower variance in their policy gradient estimates, e.g. via generalized advantage estimation (Schulman et al. 2015 As a result, the high-fidelity policy gradient estimator is typically biased (violating Assumption 5 in Section 5/, and extending our convergence analysis to this setting is significantly more challenging. Nevertheless, studying the bias variance trade-off in this regime may still allow MFPG to yield meaningful variance reduction and performance gain for contemporary on-policy actor-critic methods, and thus merits deeper investigation. Moreover, within the on-policy setting, the proposed MFPG frame- work could, in principle, be extended in model-based fashion, where the low-fidelity control-variate random variable X and its sample mean ği' are computed from learned dynamics model.
Another important direction is to extend MFPG to off-policy settings. On the one hand, the MFPG frame- work might be adapted to iiportance-sampling-bess
off-policy policy-gradient estimators, e.g. by replacing estimators. In such scttings, variance reduction arguably even more critical than in on-policy policy-gradient moth- ods (Metelli et al., if one wishes to extend an MFPG-type approach to modern off-policy algorithms such soft actor-critic (SAC) (Haarnoja et al. 2018), developing multi-fidelity control variate techniques for off-policy temporal- difference (TD) learning would be an important step. This extension is less direct. In contrast to on-policy policy-gradient methods, where high-variance Monte Carlo returns motivate sophisticated variance-reduction techniques, most modern off-policy algorithms,, such as SAC, rely on one-step TD learning and experience replay, which tend to produce relatively low-variance value targets. Their main challenges are primarily bias- related, e.g. value overestimation (Fujmooo et al. 2018). distribution shift (Kumar et al., 2019). and bias accumulation over long horizons (Park et al.) 2025). Even so, multi-fidelity control variates may still help reduce noise in bootstrapped targets and the variance of gradient estimates, e.g. in the policy-improvvemett (policy extraction) steps. In particular, pathwise gradient estimators used in TD-learning bearning-bbsedd approaches can still suffer from high variance, depending on the conditioning and smoothness of the learned value function MMhamedd et al. 2020).
the high- and low-fidelity random variables with off-policy, seportance-weighree
2020), ; making multi-fidelity control variates particularly appealing. On the other hand,
A separate line of future work is to extend the multi-fidelity control variate technique to incorporate offline datasets. One scenario is that a large volume of low-fidelity data is available as a static offline dataset, while smaller amount of high-fidelity data can be gathered online. In principle, one could develop mechanism similar to the MFPG framework in this work for incorporating such offline (and therefore off-policy) low- fidelity data in a way that aims not to introduce additional bias into the policy gradients estimated from online high-fidelity samples. Concretely, the low-fidelity sample mean á would be estimated from an offline dataset. If, in addition, the offline dataset logs (or allows reconstruction of) the noise samples used for action selection, then one could draw correlated high-fidelity trajectories in the high-fidelity environment via the reparameteriization scheme introduced in Section 4 and use the offline low-fidelity trajectories to construct control variates. Another scenario is the standard offline RL setting, in which the high-fidelity samples themselves come from an offline dataset that may be augmented with online low-fidelity simulation data. Extending MFPG to this sctting is less direct. The challenges are at least two-fold. First, the specific MFPG framework studied in this work requires the high-fidelity random variable x to be computed from on-policy samples, whereas offline data are inherently policy. One might use importance sampling to reweight the offline data and then use online low-fidelity simulation data to reduce the variance of the resulting importance- weighted high-fidelity fidelity gradient estimates. Second, extending multi-fide fidelii-iiiity control variates commonly used, ID-llerning-basee offline RL algorithms (Kumar et al. 2020: Kostrikov et al. 2021 faces challenges similar
41

---

Published in Transactions on Machine Learning Research (02/2026)
to those discussed above for off-policy TD methods: the dominant issues bias-related, e.g. inaccurate value estimates for out-of-distriution actions and severe distribution shift between the behavior and target policies (Levine et al. 2020
42


Reviews:
[{"review_id": "UtQCO6AgUG", "reviewer": "Reviewer_hz18", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary\n\nThe paper _A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation_ introduces a new variance reduction technique for policy gradient methods when (1) a simulator of the environment is available and (2) data can be gathered from this simulator cheaply. This variance reduction technique uses correlated trajectory samples between the simulator and environment to construct control variates to reduce the variance of the policy gradient proportionally to a function of the Pearson correlation between these trajectories (Equation (1)). The paper studies this variance reduction approach applied to REINFORCE, calling the resulting algorithm the _Multi-Fidelity Policy Gradient_ (MFPG). The paper provides both theoretical justification for MFPG, including convergence to first order stationary points, as well as an empirical investigation of MFPG on a number of continuous-control problems.", "claims_and_evidence_yn": "No", "claims_explanation": "# Main Argument\n\nOverall, the paper motivates the use of low-fidelity data for variance reduction in policy gradient well. The developed algorithm is sensible and theoretically justified by providing improved convergence rates over REINFORCE to a first order stationary point with correlated data between the environment and low-fidelity simulator. Yet, I am concerned about the empirical study of the MFPG algorithm and that the claims made in the paper are not supported by rigorous empirical evidence. \n\n**Too few random seeds**: My main concern is that only 5 experimental repetitions were used throughout the empirical study, which is likely far too few for a rigorous statistical characterization of the algorithms considered [1,2,3]. That reinforcement learning (RL) algorithms exhibit high variance performance distributions is well-known, and recent work has shown that performance measures based on only a few experimental repetitions can be erroneous or confidently incorrect [1,2]. I expect that the reported results would change with more experimental repetitions.\n\nThe paper also concedes to this. On page 10, paragraph **Result reporting**, the paper states\n\n> Since five-seed results remain noisy... we report median, maximum, and minimum values.\n\nI would point out that simply reporting *different* performance estimates does not circumvent the problems inherent to using few experimental repetitions. In fact, it is not entirely unlikely that these reported statistics are not representative of the true values of the underlying performance distributions. This complicates an accurate evaluation of the claims made in the paper. For example, in the discussion relating to Figure 3, multiple claims are made regarding the median performance of MFPG and other variance-reduction methods in comparison to each other and to the High-Fidelity Only baseline. But, without an accurate estimate of the true median performance of all these algorithms, these claims are difficult to verify.\n\n**Statistical confidence is not reported and reported errors bars overlap**:  The paper does not report any notion of statistical confidence, likely due to the use of only a few experimental repetitions. Without any measures of statistical confidence, deciphering statistically significant differences in algorithm performance is difficult if not impossible, complicating an accurate evaluation of the presented experimental results. \n\nInstead, the paper replaces notions of statistical confidence with maximum and minimum performance. It is not entirely unlikely that the Student-T confidence intervals for the data presented are of comparable magnitude to the reported max/min regions, and these kinds of confidence intervals would likely be optimistic estimates of confidence [1]. Assuming this is true, the paper then makes multiple claims where estimates of statistical confidence overlap. Even if this assumption is not true, the paper makes claims where the reported performance distributions do overlap, often severely. \n\nI will list a few examples next, noting that in all these examples, the performance distributions overlap often severely throughout the experiment. This perhaps indicates no statistically significant difference among algorithms, though this could be remedied with more experimental repetitions.\n\n---\n\n**Example 1** On page 12, in relation to Figure 3, as noted above.\n\n**Example 2** On page 13, in discussion of Figure 5:\n\n> the median performance gains of MFPG over High-Fidelity Only are greater when the correlation is higher (smaller gap).\n\nIn this figure, the reported performance distributions overlap nearly entirely throughout the experiment. Further, it actually seems that the benefit of MFPG over High-Fidelity Only is greater under 2x gravity (lower correlation) than 0.8x gravity (higher correlation), which seems to contradict the statement above. With more random seeds and more rigorous notions of statistical confidence, the relationship between performance and dynamics shift for these algorithms will likely be clearer.\n\n**Example 3** On page 16, in discussion of Figure 10:\n\n> when the dynamics gap is mild... reparameterization substantially accelerates learning\n\nBut again, we see that the performance distributions overlap at multiple point at the beginning and throughout the experiment.\n\n**Example 4** On page 16, in discussion of Figure 11:\n\n> In practice, we observed that periodically reconstraining (the correlated) low-fidelity rollouts back to their high-fidelity counterparts can strengthen correlation and improve performance\n\nBut again, the performance distributions overlap severely throughout the experiment.\n\n---\n\nWhile I recognize that the use of relatively few seeds is a common practice in deep RL, this norm does not reflect scientific rigour. I would urge the community－and this paper in particular－to aim to provide rigorous empirical evaluations of algorithms with high statistical confidence. My recommendation is:\n1. Report rigorous statistical characterizations of confidence for which all assumptions are satisfied. \n2. Use many more experimental repetitions (e.g. ≥ 20) so that the reported levels of confidence are accurate \nBootstrap confidence intervals tend to be a good choice for (1), as they make no assumptions on the distribution of data (unless your dataset satisfies the assumptions of another confidence measure). \n\n\n# References\n\n[1] Andrew Patterson, Samuel Neumann, Martha White, Adam White. _Empirical Design in Reinforcement Learning_. JMLR, 2024.\n\n[2] Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer. _How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments_. arXiv preprint, 2018.\n\n[3] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. _Deep Reinforcement Learning that Matters_. AAAI, 2018.", "audience_yn": "Yes", "audience_explanation": "The paper presents a novel method for variance reduction in policy gradient, which is of interest to much of the audience of TMLR.", "requested_changes": "I would be happy to re-evaluate my score if my concerns in my Main Argument above were addressed. In summary, I am concerned that the empirical study is lacking statistical rigour. \n\nThe paper would need to provide a more rigorous statistical characterization of the MFPG algorithm, including increasing the number of experimental repetitions and reporting methods of statistical confidence for which distributional assumptions are satisfied (e.g. bootstrap confidence intervals). This could also include reporting more aggregated performance statistics, as the paper reports when analyzing the aggregate performance collapses across all experimental settings.", "broader_impact_concerns": "None", "additional_comments": "# Small Things\n\nThese did **not** affect the scoring of the paper.\n\n1. In Figures 3 and 4, the colours for *DARC* and *Low-Fidelity Only (100⨉)* are hard to distinguish\n2. In the caption to Figure 3, the following statement is made\n   > Under large dynamics gaps, all multi-fidelity approaches converge toward the High-Fidelity Only baseline performance\n\n   The term *converge* is a bit loaded, and is likely not what is meant here. \n3. Section 6.2: \"... that also use state-value function subtraction an an effort to variance reduction\" → \"... that also use state-value function subtraction an an effort to _reduce variance_\"\n4. The term _performance collapse_ is used before it is fully defined\n5. The symbol $X_{\\tau^l}^{\\pi_\\theta}$ is confusing. At the outset, it is remarkably similar to the symbol $X_\\tau^{\\pi_\\theta}$ which is defined as $X_\\tau^{\\pi_\\theta} \\doteq \\frac{1}{T}\\sum_{t=0}^{T-1} G_t \\log \\pi_\\theta(a_t \\mid s_t)$. I assumed that $X_{\\tau^l}^{\\pi_\\theta}$ was exactly this value for the low-fidelity simulator, which seems to be incorrect since later $X_{\\tau^l}^{\\pi_\\theta}$ is more or less defined as an arbitrary, trajectory-dependent control variate (paragraph 2 of section 4), which is very confusing. For example, later $X_{\\tau^l}^{\\pi_\\theta}$ has the exact form of $X_\\tau^{\\pi_\\theta}$, except that the advantage function of $\\pi_\\theta$ in the environment (not the low-fidelity simulator) is used, which is clearly different from the advantage function of $\\pi_\\theta$ in the low-fidelity simulator.\n6. Some of the reported results are a bit surprising, without much discussion. For example, in Figure 3, we see many of the algorithms attaining higher performance than the _More High-Fidelity Data (15x)_ algorithm. In the top two subplots, we see that often the _Low-Fidelity Only (100x)_ algorithm is competitive to the _More High-Fidelity Data (15x)_ baseline. This is quite a surprising result since the _More High-Fidelity Data (15x)_ algorithm was considered as the _oracle_ baseline, outlining the performance we would like to achieve if we had access to lots more data. The paper would likely benefit from a more in-depth discussion of this.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ebq1ub9e6m", "reviewer": "Reviewer_bHiF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Multi-Fidelity Policy Gradients (MFPG), a reinforcement learning framework that efficiently leverages abundant, cheap data from low-fidelity simulators alongside a small amount of high-fidelity data to create an unbiased, variance-reduced policy gradient estimator, enabling more sample-efficient training and robust performance even with significant dynamics gaps. The authors also provide some theoretical analysis of the MFPG.\n\nThe contributions are: (1) the MFPG framework for constructing unbiased, low-variance policy gradient estimators; (2) a practical multi-fidelity variant of the REINFORCE algorithm with proven asymptotic convergence guarantees and improved finite-sample convergence rates; and (3) empirical validation demonstrating MFPG's performance gains, robustness to dynamics gaps, and effectiveness under reward misspecification across simulated robotics benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide some theoretical analysis (though with some comparatively strong assumptions) to validate the proposed MFPG framework, which is commendable. Furthermore, the authors conduct numerous experiments in Section 6, demonstrating the effectiveness of their approach against some prior methods. It should also be noted that the authors tune the hyperparameters of baseline methods instead of reporting the raw performance without any tuning. This highlights that the advantages of the proposed MFPG framework when compared against baselines.", "audience_yn": "Yes", "audience_explanation": "This paper mainly discusses off-dynamics RL. This field is strongly related to reinforcement learning, and to the best of my knowledge, TMLR has accepted numerous RL papers. Second, this paper is related to policy adaptation in RL, which should be interesting to numerous RL researchers, especially that there are numerous off-dynamics RL papers accepted in top-tier AI conferences like ICML, ICLR, and NeurIPS", "requested_changes": "1. I think the authors have included numerous off-dynamics RL papers as references. However, since this is a growing area, this paper can still benefit from including more related off-dynamics RL papers [1-3]. Also, there are numerous off-dynamics RL papers submitted to ICLR 2026\n2. I understand that this paper mainly focuses on the online off-dynamics RL setting, but it still raises questions on whether the proposed method can be extended to other scenarios, e.g., the source environment is offline while the target environment is online. I do not mean that the authors should conduct experiments for this, but I think it would be better to discuss this in the revised manuscript. That being said, I would like to see some clarification on how the proposed method can be extended to other scenarios. If it could, how, and if it could not , why.\n3. The authors provide a long analysis on Page 6, but comment that \"However, practically, we only explicitly sample values for the policy outcomes\". That kind of makes the previous analysis and statement weak, posing a gap between them. It would be helpful to clearly state the implementation choice and the gap between the analysis and the practical implementation\n4. Assumption 5 in the theoretical analysis part is a bit strong. When the high-fidelity data is scarce, how can one ensure that the gradient is unbiased? It would be better to clarify this either in the main text or the appendix\n5. The authors comment that they \"construct an unbiased, variance-reduced estimator\" in the abstract part, but there is no corresponding theoretical analysis of these properties. There is also no empirical evidence on the unbiasedness. The authors should fix such statements\n6. The authors comment that \"To isolate the effect of their multi-fidelity mechanisms from the backbone algorithm and enable fair comparison, we adapt both PAR and DARC to the REINFORCE backbone\". It raises questions on whether the proposed MFPG framework can be extended to the scenario when SAC is adopted as the backbone. The authors do not need to conduct experiments on this, but should add some discussions either in the main text or the appendix\n7. Figure 9 shows the parameter study of the introduced hyperparameter $\\eta_{\\rm ma}$ for the moving average. It turns out that the MFPG framework can be sensitive to different choices of $\\eta_{\\rm ma}$, while the authors comment on Page 10 that \"and found the performance to be insensitive to this choice\". This should be clarified and fixed if necessary\n\n## References\n\n[1] Cross-domain offline policy adaptation with optimal transport and dataset constraint. ICLR 2025\n\n[2] Policy Learning for Off-Dynamics RL with Deficient Support. AAMAS\n\n[3] MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning. ArXiv", "broader_impact_concerns": "I think the authors fully address the potential broader impact concerns, and they include the broader impact statement in the manuscript", "additional_comments": "There are some typos in the manuscript, e.g.,\n- On page 5, $\\gamma \\in(0,1]$?\n- On page 8,  “goodness\" \n\nI do not have further comments. Overall, I think this paper is solid and the requested changes are generally easy to resolve. I recommend a clear acceptance of this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "vBfSZYKmie", "reviewer": "Reviewer_KL6A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThe paper introduces a new way to use data from a low fidelity simulator to augment learning in a higher fidelity environment.  The method uses the data points from the low fidelity simulator not to change the target of the policy gradient algorithm, but to reduce the variance in the update steps.  It does so by adding a ”control variate” term representing the variance to the target function and then weighting that term involving the estimated variances so that the result remains unbiased.  A theoretical result to that effect (and the corresponding convergence result to a local optima) is proven.  Empirical results are then provided comparing the method to two earlier methods that attempt to use low fidelity values in gradient updates by changing the target as well as baselines with only high or low fidelity data.  The new method generally performs on par with the previous methods but is able to succeed in domains where the rewards are anti-correlated between the simulators as well.\n\nStrengths of the paper:\nThe theoretical results appear correct in the case where variances are exactly known.  I have some concerns about the practical implementation but that doesn’t affect the theoretical result.\n\nThe method seems, to me, to be one of the “safer’ uses of lower fidelity data in policy gradient because it manages to not disturb the convergence properties, and has relatively fewer hyper-parameters than other methods.  The empirical results indicate the sensitivity to what hyper-parameters the new method have are not a big concern.\n\nI really like the result in the anti-correlated reward case.  I’m not aware of any other method that would succeed here because they usually assume a positive correlation or at least some sort of bounded dominance between the simulators, but here the math works out that the method can correct the anti-correlated term and use it to its advantage, which seems useful.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper makes three main claims and I think two are fully are supported but one of the claims (claim ii) needs to either be made more specific or require more evidence.  In addition, there are statements in the paper that could be construed as claims that need to be cleaned up. See below for details.\n\n1) Variance reduction compared to state-value subtraction.  This seems to be largely covered in the theory and the experiments in Section 6.2\n\n2) The new method outperforms a baseline using high-fidelity simulator data and matches the strongest multi-fidelity approaches.  This claim is partially supported.  I agree the method is outperforming using only high-fidelity data but the small number of replicas in the experiments (see additional comments in sections below) makes it really hard to say anything (even being on par) between the MF methods.  In addition, the paper does not compare to one of the main classes of algorithms they cite in the related work (those that change exploration or simulator usage on the fly) so I think the wording of the claim should not compare to “matching the strongest multi-fidelity methods” but instead \"the multi-fidelity methods that make policy gradient adjustments” or \"Off-Dynamics RL methods\" (the authors' term for this class of methods).  Claiming a comparison to the full set of multi-fidelity methods isn’t backed by evidence.\n\n2b) On the fewer number of collapses experienced by the new method, I agree the evidence is much stronger due to the aggregate statistics reported across experiments.\n\n3) MFPG performs well even with reward misspecification.  I actually think the authors are under-claiming here.  The method manages to deal with anti-correlated rewards between simulators, and and I would add that specific example to the claim itself.\n\nIn addition to the claims above, the authors need to be more careful about certain statements in the paper that sound like claims but are actually left to future work.  A casual reader might think the authors are saying that these extensions are obvious or covered in the appendix, but they are not and the authors need to be more clear that they have no evidence that the following statements are true:\n\nPage 5: We remark that… This whole paragraph should be reworked as a list of **limitations** of the current work.  There is no solution here that can deal with multiple simulators (which many MFRL methods do), deal with different state  / actions (again others do) , infinite horizons (again others do), or different discount factors.  The paragraph makes it seem like these are in play but there is no evidence here.\n\nPage 9: Similarly, the inability to deal with Ant3 should be brought up at the beginning as a constraint / limitation of this method.  If you require the ability reset the simulator to arbitrary states at any time, this should be stated in the environment descriptions up front, not relegated to a footnote in the empirical section.\n\nAll of those should be reworded to make it clear that these are **limitations** of the current approach and are left to future work without evidence presented here.", "audience_yn": "Yes", "audience_explanation": "Better use of simulated data in RL is a major problem for deploying the field in practical environments and the current method gives a way to do so with fewer hyper-parameters than other methods.", "requested_changes": "Overall this is a promising new method and I don’t think there is anything completely disqualifying in the idea or execution. However, I have some concerns that need to be addressed about (1) the wording around the limitations that I covered in the claims section, (2) the formal descriptions of the policies on each step (3) the way the omega term is specified in the algorithm / text (4) the wording / comparisons in claim 2 and (5) the lack of replicas.  Each of these seems like they could be resolved, but as they currently stand, they make the paper not fully support (or fully clarify) the claims and assumptions.\n\n(1) was covered above - just reword the pseudo-claims as limitations.  This is crucial but should be straightforward.\n\n(2) The terminology and variables used to describe policies \\pi_\\theta seem imprecise and may have implications on the theoretical results.  I am concerned the author’s formulation in equation 2 uses  pi_theta interchangeably between the old/new terms.  But \\theta is updated between old and new, so really what you have here is \\pi_theta_k and \\pi_theta_k+1, which I do see used in the appendix proof of Theorem 1, but not in the discussions in the main paper.  The authors need to fix this terminology discrepancy and make sure none of the derivation assumes \\pi_old = \\pi_new.  This is important but hopefully straightforward and would give greater confidence in Claim 1.\n\n(3) The omega term introduced in the algorithm is impractical to implement the way the algorithm is written.  I scratched my head when I saw that because there is no way to guarantee two simulators have the same noise models between two different transition functions.  The authors avoid this with a reparameterization trick, which seems ok to me, but they ought to be up-front that they are going to do so and state clearly that sampling omega is impractical.  Furthermore, the pseudocode should have the reparameterization trick in it so that future users can follow the implementation rather than trying to build the oracle function currently in there.\n\n(4) The empirical results currently only partially support Claim ii as written.  First, while the authors list several classes of MFRL approaches, they only empirically compared against two “Off Dynamics RL” algorithms, DARC and PAR, while there are no comparisons to the “MFRL” methods mentioned in the paper.  The authors state that those methods use “new training paradigms or exploration strategies”, which is true, but why can’t you compare to one of those methods?  As it stands, claim (ii) states that MFPG “matches the strongest multi-fidelity methods” but that is not the case if you’re not comparing to a whole class.  The claim could be made true by referring to “Off dynamics RL methods” or “multi-fidelity methods that adjust the gradient updates” or something more specific like that. This is crucial but could be just a wording change to the claim unless you want to add more competitors to the experiments. \n\n(5) Finally, the empirical results are all based on a small number of replicas (5 for each bar).  While reporting min/median/max for 5 replicas would be fine if the bars were clearly separated, in all cases except for the mis-specified reward one, the bars heavily overlap, and sometimes the new method has better/worse upper/lower performance than the competitors, and sometimes the opposite.  Given the small number of samples, it’s hard to say much about how these methods compare, although aggregate statistics across the domains like those you did for catastrophic failure are more reasonable (and indeed that claim seems justified). I suggest either running some more replicas or reporting more aggregated statistics so the trends here are clearer.  I also think the authors should say something about why the reduced-variance updates (from Claim 1) are not leading to any actual performance difference in the new algorithm.  Why aren’t the results coming in with lower variance or outperforming methods that have higher variance in their updates?  The crucial part of (5) though is either to add more replicas or to report more \"aggregated across domains\" statistics to increase the confidence in the claim.", "broader_impact_concerns": "I think the Broader Impact statement as written is sufficient.", "additional_comments": "Minor Points:\n\nAlg 1 – The alorithm should have line numbers\n\nPage 15 – bolded capital I in “In this section,”\n\nPage 5 – It appears you are assuming episodes are always finite horizon and that assumption needs to be formalized in the MDP description itself.  If the MDP has unbounded episodes then you need to make gamma \\in [0, 1) – that is gamma can’t be 1 as that would allow one to solve NP complete problems.  If instead you are assuming a bounded horizon, the H term should appear in the MDP description.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)