Mitigating Steady-State Bias in Off-Policy TD Learning via Distributional Correction
Authors: Emani Naga Sai Venkata Sowmya
Amit Kesari
Ajin George Joseph
Venue: Accepted by TMLR ()

Abstract: We explore the off-policy value prediction problem in the reinforcement learning setting, where one estimates the value function of the target policy using the sample trajectories obtained from a behaviour policy. Importance sampling is a standard tool for correcting action-level mismatch between behaviour and target policies. However, it only addresses single-step discrepancies. It cannot correct steady-state bias, which arises from long-horizon differences in how the behaviour policy visits states. In this paper, we propose an off-policy value-prediction algorithm under linear function approximation that explicitly corrects discrepancies in state visitation distributions. We provide rigorous theoretical guarantees for the resulting estimator. In particular, we prove asymptotic convergence under Markov noise and show that the corrected update matrix has favourable spectral properties that ensure stability. We also derive an error decomposition showing that the estimation error is bounded by a constant multiple of the best achievable approximation in the function class. This constant depends transparently on the quality of the distribution estimate and the choice of features. Empirical evaluation across multiple benchmark domains demonstrates that our method effectively mitigates steady-state bias and can be a robust alternative to existing methods in  scenarios where distributional shift is critical.

Published in Transactions on Machine Learning Research (01/2026)
Mitigating Steady-State Bias in Off-Policy TD Learning via Distributional Correction
Emani Naga Sai Venkata Sowmya
es cs19b045@iitp..c..k
Deportment of Computer Science and Engineering
Indian Institute of Technology Tirupati
Amit Kesari
cs cs196003@hittp.cc.o
Deportment Computer Science and Engineering Indian Institute of Technology Tirupati
Ajin George Joseph
ojin@iillp.ccmm
Department of Computer Science and Engineering Indian Institute of Technology Tirupati
Reviewed on OpenReview: https: opw.reviee.r..eee...m net/ forum? id- QLZANgs
Abstract
We explore the off-policy value prediction problem in the reinforcement learning setting, where one estimates the value function of the target policy using the sample trajectories obtained from behaviour policy. Importance sampling a standard tool for correcting action-level mismatch between behaviour and target policies. However, only addresses single-step discrepancies. It cannot correct steady-state bias, which arises from long- horizon differences in how the behaviour policy visits states. In this paper, propose off-policy value-pred plleeppreiition algorithm under linear function approximation that explicitly corrects dis- crepancies in state visitation distributions. We provide rigorous theoretical guarantees for the resulting estimator. In particular, we prove asymptotic convergence under Markov noise and show that the corrected update matrix has favourable spectral properties that ensure stability. We also derive an error decomposition showing that the estimation error bounded by a constant multiple of the best achievable approximation the function class. This constant depends transparently on the quality of the distribution estimate and the choice of features. Empirical evaluation across multiple benchmark domains demonstrates that our method effectively mitigates steady-state bias and can be a robust alternative to existing methods in scenarios where distributional shift is critical.
1 Introduction
In the reinforcement learning (RL) setting (Sutton & Barto 2018 Bertsekas 2019 Meyn 2022). an agent learns to interact with an environment to achieve a goal or maximize its cumulative reward by performing specific actions and receiving feedback from the environment the form of rewards. The agent sequentially refines its behaviour using the data generated by its interactions, making RL dynamic and adaptive learning framework. The central proposition in reinforcement learning is the ability to observed data about earlier decisions and their rewards to conclude how alternative decision policies could perform and update their course of action. RL has been applicd widcly, including in game simulations (Silver et al. 2018 robolics, autonomous driving (Kiran et al. 2021), medicine (Yom-Tov et al., 2017 Tejedor et al. 2020) and communication systems (Huang et al. 2019). In these domains, agents learn strategies by exploring and refining their decisions over time. For example, RL agents in game environments achieve superhuman performance by analyzing large volumes of gameplay trajectories. Robotic systems use RL data acquire complex motor skills such grasping navigation. Autonomous vehicles learn safe and efficient. driving

---

Published in Transactions on Machine Learning Research (01/2026)
patterns by analyzing logged driving data, while communication systems optimize bandwidth allocation by reasoning over previously observed traffic and channel conditions.
In this paper, we consider the policy evaluation problem in reinforcement learning, which refers to the task of estimating the value function, which represents the expected cumulative reward from a given state following a certain policy. The policy evaluation problem has two variants: on-policy and off-policy. Tn on-policy prediction, one tries to estimate the value function corresponding to a given target policy using the sample trajectories generated using that target policy itself. However, in the off-policy variant (Baird
et al. 1995 Precup et al. 2000; Yu 2012), one intends to learn the value function using a sample trajectory generated using a behaviour policy that may be different from the target policy. The behaviour policy is the policy followed during data collection, and the target policy is the policy for which the value function is being estimated. In many real-world settings, an agent cannot freely interact with the environment to collect trajectories under the policy of interest and must instead rely on logged trajectories generated by a behaviour policy. Although this mismatch may introduce statistical challenges, also provides an important advantage: off-policy data reuse enables substantially greater flexibility. An agent can learn from trajectories produced by many different behaviour policies, allowing it to exploit existing datasets without additional interaction cost. This is particularly valuable when online exploration is expensive or unsafe-for example, autonomous driving systems that learn from human driving logs, clinical decision-support models trained on retrospective treatment pathways, or communication networks that leverage historical traffic traces. Off-policy learning also enables multiple value functions to be estimated in parallel from the same data stream, supporting scalable, model-free evaluation in large systems. Consequently, the distribution induced by the behaviour policy becomes the central object shaping the learning process. Off-policy estimation methods often rely on importance sampling (IS) (Rubinstcin 1981 Glynn & Iglchart 1989) because it is unbiased estimator. The fundamental concept behind IS (Tokdar & Kass 2010 is to correct the samples obtained from sample trajectory generated by a behaviour policy to align with the likelihood of that trajectory occurring under the targeted policy. The importance sampling approach integrated with many on-policy variants, such as gradient temporal difference {Sutton et al., 2009 Yu 2017), temporal difference with correction (Sutton) let al.] 2009; Yu 2017), and temporal difference with eligibility traces (Precup et. al.] [2001) to obtain the off- off-policy solution. However, an important drawback of this technique is its susceptibility to imprecision because of the high variance induced by the importance weights (Mandel et al. 2014) and the discropancies associaled with state appearance probabilities (Tsitsiklis & Van Roy 1997].
In this paper, we analyze the deviation of the on-policy solution from the off-policy solution due to the steady-state bias which ariscs due to the discrepancy in the steady-state distribution induced by the target and behaviour policies. When one observes the marginal distributions from the target policy and behaviour policy after a sufficiently long time (mixing time), the marginal distributions settle down to the stcady-state which is unique to the corresponding Markov chain. Steady-state bias arises whenever the state visitation distribution in the behaviour data differs from the target policy's steady-state distribution. This occurs when the trajectory provides only partial state coverage or, even when every state appears infinitely often, visits states in proportions that differ from those induced by the target policy. As a result, the average return may not accurately reflect the true expected return, which can lead to sub-optimal behaviour. Off-policy bias correction is a fundamental challenge in reinforcement learning, particularly in settings that utilize experi- ence replay or batch data from previously exccuted policies (Precup ct al., [0011] [Sutton et al. 2016). The systematic bias introduced into the value function estimatiin due to the discrepancy between the behaviour policy's ' stationary distribution and the target policy's 's state visitation distribution persists even with un- biased importance sampling corrections, as it stems from long-horizon distributional mismatch rather than single-step policy differences (Chandak et al.) 2021). This steady-state bias becomes particularly problem- atic in long-horizon tasks where distributional mismatch accumulates over time (Jiang & T.i 2016 Tang
et al. 2020). Several recent approaches have addressed this by estimating stationary distribution corrections
through marginalized importance sampling (Liu et al., [2018] 2019). dual function approximation (Zhang et al. 2020 or direct optimization of distribution matching objectives (Nachum et al. 2019 Yang et al,
2020 Frameworks like Universal Off-Policy Evaluation (Chandak et al., 2021) further improve estimation by enforcing consistency between learned value functions and off-policy estimators. These methods often formulate the correction as a minimax optimization problem over density ratios or leverage policy gradient with generalized advantage estimation (Schulman et al. 2015)
The convergence and stability of such meth-
2

---

Published in Transactions on Machine Learning Research (01/2026)
ods are closely tied to the 'deadly triad' of function approximation, off-policy training, and bootstrapping, which can lead to divergence without careful regularization or correction mechanisms (Voloshin et al. 2019 Wang et al. 2017 Yu 2017).
In this paper, we fundamentally analyze off-policy temporal difference learning by tackling the critical prob- lem of steady-state distribution mismatch. We rigorously demonstrate that long-horizon bias stems not only from policy differences but also from the divergence in how states are visited under target versus behaviour policies. Our analysis shows how such a distributional shift amplifies approximation error through the deadly triad of bootstrapping, function approximation, and off- sampling. To address this issue, we introduce dual correction mechanism. It combines standard per-step action reweighting with novel parametric esti- mation of stationary distribution discrepancies. We further show that the resulting update converges under ergodic Markov noise. We also establish that the corrected value estimates stabilize when the rebalanced Bellman operator exhibits spectral negativity. Most significantly, we derive an error decomposition showing that the total cstimation error is bounded by a constant multiple of the best achievable approximation. This constant depends on the accuracy of the distribution estimale, the conditioning of the feature malrix, and the degree of policy misalignment.
Our work distinguishes itself from prior distribution correction methods by providing a targeted solution to the persistent, problem of steady-state bias in off-poliey TD learning. While frameworks like DuaIDICE (Nachum et al.) 2019) and DIAE (Zhang et al.) 2020) address general distribution ratio estima- tion through complex dual optimization, our approach offers a direct, computationally efficient correction that integrates seamlessly with standard TD updates. Unlike policy optimization methods such as CQL [Kumar et al., 2020 OptiDICE (Lee et al. 2021) which focus on policy improvement, we specifically address value prediction accuracy under distributional shift. While (Liu et al., 2018) addressed the "curse of horizon' through marginalized importance sampling, our approach uniquely identifies and corrects for the persistent steady-state bias that remains even after one-step importance sampling is applied, offering a complementary perspective on the distributional mismatch problem in off-policy evaluation. Also Emphatic TD (Sutton et al. 2016 Yu 2015), employs recursive emphasis weighting to implicitly approximate distribu-- tion correction, preventing the deadly triad (function approximation + bootstrapping + off-policy learning) from causing divergence using emphasis weights to prioritize updates for states that are important to the target policy. Relatedly, (Hallak & Mannor 2017 tackles distribution- tionary density ratio between the behaviour and target policies, but requires solving an inherently unstable fixed-point ratio-estimation problem. Our focused approach correcting steady-state bias through explicit distribution modeling rather than general optimization frameworks providdss both thcoretical clarity and practical advantages for the fundamental problem of off-policy value prediction.
bias by learning the sta-
2 Background
The reinforcement learning setting is an optimal sequential decision-making paradigm under uncertainty characterized as Markov Decision Process (MDP) (Puterman) 2014 Bertsckas 2019 Meyn 2022), which is a controlled, time-homogeneous, stochastic process that is defined by the 4-tuple (S, A, P, Rt), where S is the state space and A is the action space. In this paper, we consider a finite state and action spaces with A x S - [0,
S= { :,.,....""" )) Here P P(s = s' 8, a, = a, S = -, = .....) = PP( = s'|s: = s, a, = a) is the probability that the next state is s' conditioned on the fact that the current state is 8 and the current action is a. Additionally, the reward function R: S x A x S + R assigns a numerical reward to each transition. P and R. define the dynamics of the stochastic system. At each instant, an action is chosen according to a stationary stochastic policy ग S x A + [0, 1], where (-1s)) is a probability mass function over the action space A conditioned on the state 8 E S.
the probability transition function, where P(s, a, 8') =
In this paper, we consider the prediction problem reinforcement learning, which defined as follows: For a given target policy π and discount factor 7 E (0,1) 1) (that represents the agent's preference for immediate rewards versus future rewards), the goal is to evaluate the value function V, E RR" associated with the target
3

---

Published in Transactions on Machine Learning Research (01/2026)
policy which is defined as the expected long-rum y-discounted cost:
Vi(s)=
[(T)]s s) T-E
E
(1)
where R(r) = - R((s, a 8(41), with 8; represents the state at instant I, a, ~ x+1ss) represents the action chosen at time and 8141 P(st, a);:) represents the next state. Note that the above definition is well-defined as 7 E [0, 1) and by appealing to the bounded convergence theorem.
The value function in vector form is expressed as Va [Vr(s"),, (=*), [T") E R". The value function Vz satisfics the Bellman equation: V- TVV7, where Tr 1 - R is the Bellman oper- ator with TUU = Rs + +PPUU Here, Pr E IR*x with [Pr]as' = Eeea T(als) a, 8') and (s)s) = Eves Eaea a(s,a, R') a, s') is the one-step average reward. From the Bellman equation, one can directly compute V = (I- (1-7Px) Rg whose computational complexity is dominated by the matrix inversion (O(n",, c 2.374). In the RL sctting, the model parameters P and are unknown, and one seeks learn the value function Vz under the generalive model setting, where a realization of the stochastic process in the form of an infinitely long sample trajectory So, T1, S1, a1, S2,...... is available, with so Po (Po initial distribution), a, ~ T( (-1ss),, S141 P(s,, ac. and rz+1 = R(ss, 8;+1).
Temporal difference (TD) learning (Sutton & Barto 2018) is the classical approach for the prediction prob- lem, where the value function V, E R" is iteratively updated in the direction of the temporal difference T++1 + (++++1) - (s:).. However, when the state space is large, this method suffers from the curse of dimensionality (Tsitsikiis & Van Roy 1997: Sutton Barto 2018). To overcome this, one effective strategy is represent the value function in lower-dimensional subspace, thus reducing computational and storage demands. Here one approximates Va using linear function approximation by projecting it into the subspace {Ør x E R C R", where & < n (Titsiikis & Van Roy 1997). The feature matrix & contains basis functions that capture the critical characteristics of the state space. This projection not only renders the learning process more tractable but also preserves the essential dynamics of the original high-dimensional problem.
((x1)
(2)
o(s") n
where, (ss) - (2)) (s))) E R is called the feature vector associated with the state s S and di 5 R are feature/ /basis functions. The most commonly used parameterized basis functions include radial basis functions (RBFs), polynomials, and Fourier basis functions. Radial basis functions are typically expressed in Gaussian form: (5(s) = exp(-(2o?)- - pill?), depends solely on the distance between the state and the centre 14, relative to the feature width, di, with a parameter size of the order (k)).
In this paper, we consider the off-policy variant of the prediction problem (Precup et al. 2001 Sutton & Barto 2018), where one socks to estimate Vn: using a sample trajectory, where action at every instant is generated using a behaviour policy na that may be different from the target policy æ. This implies that for the given infinitely long sample trajectory To So, ag; F1, $1, a1, r2, 82, a.,..., we have So Po, a, TT(-(s)); S ~ P(st, ar,,,ar,)) and r++1 R(s,, a $t41).
Assumption 1 (Ergodic Behaviour Policy). The Markov chain (si induced by the behaviour policy satisfies:
(1) Irreducibility:: &s,8 E S, 31 E N such that Pt (s, > 0.
(1) Aperiodicity: The greatest common divisor of ( P (s, 0) is for every E S.
Consequently, the choin admits unique stationary distribution vp with vb(s) 0VS S and vj Pr, =n- Assumption 2 (Feature Independence). The feature matrir T E Raxk satisfies rank($) k. implying
mmin T T ④) > 0, ker(+))=10 = (0),
where min denoles the minsmum singular value and > O denotes positive definiteness.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Assumption 3 (Coverage). The behaviour policy Tu dominates π in the Rodon-Nikodye sense:
((, ES x A, T<a 8) > = mb(a 8) 0.
π (a St)
Equivalently, the importance ratio Pr.
is almost surely bounded: supt Pr. < oc.
Tb (ar St)
In off-policy linear function approximation, one projects the value function V onto the column space of Φb Tsitsiklis & Van Roy 1997
wt arg min ||V owlle.
(3)
WER
where the weighted norm is defined 1 Dí wwl. Here, V, is the unique stcady-state distribution of the behaviour policy's Markov chain (i.e., (s) lim P(s: s) and vi Pr.. 11, Since {&w
w E RR*} is closed and convex, unique w" exists ( has full column rank), yielding the approximation
(s) ~ o(s) w² for all s. This optimization is solved by the off-policy TD update (Precup et al. 2001 2000
W++1 W. aepe (rtt + 70(8++1) Tw. ((s:) (w,s)))))))))) (w:)((s))
with the importance sampling ratio P [;))) which corrects for the policy mismatch in the behaviour data. The limit point wifh of off-policy TD learning with linear function approximation is characterized by the fixed-poite point equation Yu 2012
Em pP2) TD = En
(4)
which represents projection of the Bellman equation onto the feature space weighted by the behaviour policy's stationary distribution. This solution constitutes the best approximation within the function class that satisfies the Bellman residual minimization under the bchaviour policy's steady-state distributional mismatch, rather than the target policy's natural state visitation pattern.
V
π
steady state bias
(Twhe las RR}
On-policy TD solution dhr TD
Dw off Off-policy TD solution
argmin I|V Turll
argmin V ow
On-policy projection
Oll-policy projection
Figure 1: Illustration of steady-state bias in off-policy prediction: The mismatch between behaviour policy's 's steady- state distribution vs. and that of target policy causes persistent prediction error, after one-step importance sampling correction
5

---

Published in Transactions on Machine Learning Research (01/2026)
To establish further theoretical guarantees for the off-policy TD method with linear function approximation, we first analyze key properties of the value function operator under the behaviour policy's stationary distri- bution. The following lemma quantifies fundamental operator norm bounds that govern the propagation of approximation errors through the Bellman operator.
Lemma 1.
Let vs be striclly positive probability distribution over states, Pa Markov transition matric induced by policy a, and y E (0,1) a discount factor. Then the Vo vy-weighted operator norms satisfy:
Palll, I vris and -1--9P0 - 51+7/
(5)
where ||Al|m. = supx#00
|x
and | = E. ((s)x((3)) with and the distribution mismatch coefficient
s)
Kb = max E'ES
vo()')
Proof. Let mb(5') = E, (c())s ()). Then,
|Pa Evsos) Σ Pold' (P((011)((6)) 2
En(ss) EPr(s s)x(s')2 (Jensen's inequality)
Ex('') Σ (o(s) (s)
Σ x(8')PPub()) Σ (o(ss)x((s (x(ss)x(s))22mnl vo(s')
S Kb Σ = where Ke max Mb(s') vs(
Thus we have the following operator norm bound:
P Ilvs
Pall, = -sup x
S vkb
|| |x||1
Now for the composite operator, we get
((T=>P>)>\le
< |x|||0g +y |PaxXlyn
S + 7v/))/=))ll
S <lIx|lu
Svmol|xile,
+ - 7Pllls 5 $1+7VR
Central to the above result the distribution mismatch coefficient Ka, which captures the maximum density ratio between the next-state distribution induced by the target policy and the stationary distribution of the behaviour policy. We now characterize the asymplotic approximation error of the off-policy TD solution under linear function approximation. The following theorem establishes bound on the error | off Vz lm of the TD fixed-point solution relative to the fundamental approximation limit || - Vzllus
TD
Theorem (Error Bound for Off-policy TD). Under Assumptions and negative definiteness of Ao & T (T - ", the solution wTD salisfies:
(nav ( (max, 8/(5))3/2(1 T +1) 1,ow - -Vall.,
TD off @
Vx V,, s
Amiul (-in(-Ao)) Ao) /min, vs(s)
min,
where o ( Amaz ( TT) and Ev, diag(us)-

---

Published in Transactions on Machine Learning Research (01/2026)
Proof. From
have
4 E,,
- (7p) w TD = = =m Rg.
(6)
Also, the true value function V satisfies the following Bellman equation:
1 = Rg + +VPVV
(7)
Now we bound || off Valles. Let w be the best linear approximator under 1/. Then,
w argmin fw V=
that fw- I,, Vu,, the projection of V onto the column space of & under the 19- weighted norm. The error decomposes follows:
dw
TD off
V = (frwoll TD kww) + (fw" - Va).
(8)
Hence,
@w off
V Valleg 1 @ow TD @w" ll., + ||
Vallo,
(9)
We define the approximation error Eapprux term @w 2ww'll, in terms of ||Sapprxxllv; Note that both ow TD and ow lie in the column space of
Fw"
Va. To prove the claim of the theorem, bound the
Φ. The vector Twiff satisfies
+ Evn - DWTD ② T R.,,
(10)
whereas the projection ow" satisfies
+ JE @w" Vx() 0.
Also, by multiplying the Bellman equation
by + T Eun we obtain
s E V=
Φ Eu, R. + +, E,, P.V=
T (1 - >P>>V x EE,
(11)
Combining 10 and 11
we get
① Em (1 7P') 1 Woff + TE,, (1-7Px) Vn
TD
+ Φ Ey (I (Pt) w TD - VVa) =
Φ Em, (I-7Pt)
0,
(12)
where e we obtain
& w TD
Vg Further, from 81 we have e = + (w.1l w') + Sapprox. Substituting this above,
Φ Ev, (I - 7P==) T
TD T off
w")
Eapprox
0
1 E (1 7/ Px) ) 1 woll w") + Φ Eyn
TE (I P
Eapprox 0
Φ E (1 - 7 Px) $b (w TD w') = -
E (T - 7 Pa Eapprox
Ao w of
TD
w") 4 Emg (I PP)) Eapprox.
(13)
Now,
| Tx, (1 Px) )
< | | || ((1-7P') >
(14)

---

Published in Transactions on Machine Learning Research (01/2026)
where || is the spectral norm. But, | max, vs(s,, and
[I
|
Σ yPa Eapprox (s)²
Σ min, vs(s) vo(s) (T - -p >))p))))ppee Eapprox (s)
-
11 7p P
(15)
min, (b(8)
Hence,
1 1+ 7 Rbb min, vo(s)
||(T pp aaprrox
|Eapprux
and therefore
max, (s)))
④ T (I yPr >Pp))apppox)
(1 + (1+%/bb) |aapprox|Mmm
(16)
min, (s)
Note that ④ = |0| ||. and the spectral norm of o is the large singular value. Let Fmax Fmax($) = |$|1. Also
note that, since Ag is negative definite, we have
|| S
(17)
X Amiu (~Ao)'
where Amin (-oo) is the smallest eigenvalue of -Ao Combining
13
14
16
and
17
we get
max (s)
Tmax (T)
w
TD off
w²1 S
(1 + 7 /bb |aauprox Iluy
(18)
Amiul -Ao)
min (s()
For the projected error in the mo-norm, have
| (P(wT) TD off w² w')ll,, < +
I w'll
(19)
where |1,, the operator norm of from the Euclidean space the vg-normed space. Specifically,
|| w TE dw Amax ($ T Eu, w||
|| S v Xmaax (T TE, T $).
(20)
Note that & TE,, 1 is a k x k matrix, and its largest eigenvalue is at most max, vo(s) Amax ( (T)), because Em, & max, VI(s). And Amax p T = nax (Φ). Hence,
TD (
off
llus S max ma) /ax (s) || off
w*ll
(21)
Hence from
9
18 and 21
we get
Tmax (1)) (1+7)5))
+1) |
|| [ V- Illes < [@ w
w + || S
/
min vn(s)
The above theorem bound reveals three critical components that affect the off-policy TD convergence: First, the a (max, v, (s)))3/2 term exposes the sensitivity feature scaling and distribution skew, which shows that even optimal representations suffer when vs is non-uniform or features are poorly conditioned. Second, the (1 + 7/))) factor quantifies how policy divergence (Ks. 1) amplifies approximation error through
8

---

Published in Transactions on Machine Learning Research (01/2026)
temporal credit assignment manifestation of the deadly triad where bootstrapping, function approxima- tion, and off-policy sampling interact destructively. Third, the dependence on Amin(-A.) formalizes the hardness of Bellman inversion under distribution shift, as As becomes ill-conditioned when the behaviour policy's transitions poorly align with the target dynamics. This provides a closed-form. charactorization of deadly Iriad interactions in ofl-policy TD convergence. The bound exclusively characterizes the fundamental approximation error of the asymptotic off-policy TD solution, isolating from transient algorithmic effects. When x =& (Kr = 1), the bound simplifies to the on-poliey case, but the exponential scaling 7 7Kbb explains the severe degradation under policy mismatch.
Theorem 2. Under Assumplions
and
R6?? < 1 then No is negative defonile.
Proof. For any w 00, let dw. By Assumption
+ 0. Consider the quadratic form:
w T A. w "T=== Eu (7 T)
= u LuTE,, Ev (yPa - 1)u
y u Ev, P Em
(22)
Q2
Now Q2 =u= = u E. (())u(()) = -Ilulie > 0. Since 0 (ergodicity) and u o 0, we have
Q1 u E, P=u= E Σ (((())())(PP
Σ vs(s)u(s)( Σ (d(((ss)u(s')
< Jules P (by Cauchy- Schwarz inequality)
(23)
Further, by Lemma we have
| = Kib Hollu
(24)
Therefore, from 23 and
24
we have
Q1 S 1111111 ||ules PP S VKluul
Substitute into 22 to obtain
w T A,w S TE 12 - -11u11,, = (7 - 1) ||ul
(25)
Since ||u|| > 0 and 7 - < 0 iff Kh < + we have:
w A.w S (r//kb - 1) ||ll <0 when n672
Equality holds only = O, proving Ao is negative definite.
Corollary 1. When æ = Tb. Ao is negative definite for any Y <
Proof. When π = Tb; we have Ny = Then:
w T Aw S (y 1)][v/13,
<0 Vw +
Theorem 2 establishes a fundamental condition for convergence in off-policy temporal difference learning: when the product of the policy alignment constant gら and the squared discount factor 72 less than onc, the critical matrix governing the TD update dynamics becomes negative definite. This condition, kEY provides profound theoretical insight into the feasibility of off-policy learning. The policy alignment constant

---

Published in Transactions on Machine Learning Research (01/2026)
Ko quantifies the maximum discrepancy between the next-state distribution under the target policy and the behaviour policy's stationary distribution (v)). When No is large, it indicates significant distributional mismatch; certain states are visited much more frequently under the target policy than would be expected from the behaviour policy's steady-state distribution. The theorem reveals that such mismatches become increasingly problematic as the discount. factor Y approaches 1, explaining why long-horizon tasks with high Y values are particularly challenging for off-policy mothods. Notably, when policies are identical (I Tb); we have K₁ 1, and the condition simplifies to 7 < 1, which always holds for standard MDPs. However, as policy dissimilarity increases (x₂ 1), the allowable discount factor must decrease to maintain convergence guarantees. This theoretical boundary precisely characterizes the "deadly triad" interaction between function approximation, bootstrapping, and off-policy learning, and directly motivates the steady- state bias correction.
3 Our Algorithm
Here, we propose double correction approach to address the discrepancy introduced by the steady-state distribution of the behaviour policy in the solution of the off-policy TD algorithm by effectively reducing the policy alignment constant through distributional reweighting, while simultaneously incorporating per- step policy mismatch correction Pe. To achieve this, we employ the importance sampling method to the existing off-policy TD method with one-step lookahead, where a separate state distribuito reweighting factor (ss)/h(s; 0", X²) is tied to the existing off-policy TD recursion. Here q(.) is the design probability distribution to which the solutions are guided, and ln(:; 0*, X*) = Xigo; (.) (-) a surrogate probability mixture distribution chosen from parametrized family of distributions {ga : RR + R10 E e g(z)dx = 1. 2 0} which best approximates the stcady-state distribution of the Markov chain induced by the behaviour policy with respect to the Kullback-Leibler divergence (monent projection).
0 x+
arg min Daa (vollA1g0, +... + )egoe), subject to a,ce, 1
Σ A = 1=1
(26)
where (fllg) = B, [og f(x g(x)
To model the component distributions go efficiently, we employ a flexible and analytically tractable paramet- ric family. One theoretically well-founded choice is the Natural Exponential Family (NEF) (Brown 1986). The NEF is a class of probability distributions which provides a unified framework for probability distri- butions through its canonical form that encompasses many commonly used distributions such as Gaussian, Poisson, and Bernoulli distributions, among others. The NEF has several desirable propertics, including a closed-form expression with convex tng-partiioon function, which simplifies the computation of the impor- tance sampling ratio and allows for efficient parameter updates during the learning process. A parameterized family {ge / C R²} is called a natural exponential family if go (x) = exp (0" T(x)= K()),, where I', R2 + R² and K ; R + R are continuous functions with e = {0 E R [() < c.). Note that K(0) is strictly convex in the interior of @ and V (0) = Eg [[(x]] Also, V (()) = Cov s0 [(x)] > 0. These ensure the Fisher infor- mation matrix I(0)= = (() is non-degenerate, guarantccing well-posed maximum likclihood cstimation. While all the NEF member distribbttons provide analytical tractability through their exponential structure, we employ Gaussian mixture models in our experiments for their superior approximation capabilities. We formalize this approximation and aim to guarantee that, with sufficient number of components, the KI- divergence between the true steady-state distribution and its Gaussian mixture approximation can be made arbitrarily small. A subtlety arises because the behaviour policy's 's stationary distribution vs is discrete and therefore lacks a density, making the KL divergence ill-defined. To address this, we mollify Ps by convolving it with a Gaussian kernel, yiclding a smooth surrogate. We first define the following:
For the stationary distribution of the behaviour policy vb, which is supported on finitely many states {s] s". C I2 and the smoothing parameter O, we define the mollified density as follows:
(x(x) = (v * *N (1))))))))) = Σ vo(s') ,0!,))
10

---

Published in Transactions on Machine Learning Research (01/2026)
For each > 0, Po is valid probability density, being finite convex combination of Gaussian kernels and is absolutely continuous on Rº. Further, for compact region X and clipping level > 0, we define the clipped proxy
fa(2) (x(x))x=(x) + max (z), n) 1x(x,,
(27)
where clipping is applied only on X and the density remains unchanged on X The normalizing constant is
{ Br
fn(Ø))
Zn
fn(()) dz,
(x)
Zy
Consider Gaussian-mixte mixture model (x) Aygo, (z) with X A the probability simplex of mixture weights, and each component ge, has non-degenerate covariance matrix.
mixture Theorem (Gaussian mixture approximation of the steady-state distribution). For any E > 0, there exist
> 0, < < 1, and E N such that for some parameters (6", ;²))
DRI(Po ho- S (0(n) + O(o)) log /n) + o ( + 0 (1)))
and the functions Po,X.n and x- satisfy (o.) ho x (z) M on X for some finite M.
The proof proceeds through three stages mollification, restriction and clipping, and mixture approxima- tion- follofllowed by a change-of-measure decompasition. We develop the proof through the individual results below.
Remark 1. For any bounded continuous test function f, we have
I f(x)po(z)dz Σ (o(()))B) B[f(s' +2)),, 2o 1(0,
Since 2o + 0 in probabilily, [sf(s + kx)] f(s)) by the dominaled converyence theorem, implying that pa converges weakly to va as o 1 O. This step establishes a continuous surrogate distribution for vo, ensuring the KL divergence is well-defined.
For the state space S and given R > 0, we define the compact region
XR = Uf2: I'll RR. and the tail mass as T(R, = -f Po(x) dx.
(28)
x%
i=1
Lemma 2. For every 8 E 10 and fized a > O, there exists R = R(o, 8) such that T(R, o) S 8. Moreover, for mistures of Gaussians as considered here, one can choose R so that
T TRR σ) S Cn exp
1P) 20
for a constant C' = C(p) depending only on the dimension.
Proof. Using the union bound and the Gaussian tail inequality, for each i we have
f - N(x; o o2) dæ S C(p) exp
R² 20
where ((p) is a dimennsion-dependeet constant. By summing over 1, ...... n and weighting by vs(s)) s 1, we get.
TR,o)) < C(p) exp
R²²
2
Hence, for any 8 > 0, it suffices to choose
Cn R. o1 o 2log
to ensure o(R,a) s 5.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Henceforth, we fix compact set X Xg with tail mass 7= 7(, as small as required. We do not define vero-outside renormalized density. Such truncation would make the KT. divergence infinite. Instead, we clip the density from below everywhere and then renormalize, which maintains strict positivity on IP while preserving the total mass. Note that by construction, Po.X.n strictly positive density. Further, on X we have (2) = max max (x) 2 Po x (z), and on X we leave the density unchanged, ie., fn( = (x)). Thus
by / (n(t) do / do( [/ pal(()) de, 1 (Paa(t) + 1) dæ = [1, 1+ n|
(29)
The lower bound uses Jn 2 po, and the upper bound uses Jn S Po + on X together with fn Po X Lemma 3 (L+ perturbation). There erist universal constants 01,02 such that
Po Pa,x, ll S CIT +
Proof. By definition,
/ lpo Po,x
/ Poo Po.x.ol / xo IPo Dor
IPo Po.x,alli
We now bound the integrals explicitly for each region. For all x E X we have fn(x) - Po (2) and therefore
(x. = (2(
Z.n
Hence
1po - po.x.nl = / Po Po(x)
1
1 1
/
Zn
da
Zny
xx
Po (x)
Zm T.
(30)
From 29
we have Zy E [1,1+ 1// X|]; so
12m -
< Zg - 1 S x
Zm
Zmg
Thus
lpo Po.x S ///
(31)
Xx
Now for X, we split into and B, where A= x tpo ni and B = x < n)
For A, fn(2) Po (z), (z)) = Po (z) Hence
1 Zn
lps - -o.x.nl = Po (o(
S Po (2(xx)
Integrating over A gives
J lpo Do.x.n sn/ / Po (x(x)) dr s n/Xi.
/
For E B, (x) < and fz( = 17, so
17 (o.X,((x) Zn
Thus
(p)(x) - (o)x 5 ppo(x)) + "
In + 2n.
Zn
using Zy 2 Hence
/ Ppo Po.x.nl S X|
12

---

Published in Transactions on Machine Learning Research (01/2026)
Finally, by combining the integral bounds A and B, get
Je |Pr Pa,Xo S x + 27| = 31/ 31
(32)
Combining the bounds from X 331 and X
32 obtain
p Po. x,u S X 3n| c + c2n|X),
for suitable absolute constants c1,
Lemma (KL, perturbation). There erist conslants C1 C2 misture envelope on X) such that
(depending the dimension and the
DRi(Po (o.x...) S Cin + C2rn.
Proof. By definition, we have
/ Peo logPo Po - / Po log
DRI(PO (1,x).))
We split the domain into three parts: (i) Xn (po 2 nti (ii) x (po < n); (iii) Xe and explicitly develop bounds for J Po log Po,X.n for each region as follows:
(i) Region n {po 2 n). Here fn = Pos so Po.X.n = Po Zn, and
J xn((.2n) Po log
Po po/Zn (ol)2,))
Sxm
Po log Zy S log + n|x1) s en,
for some constant
(ii) Region X n (Po < n)- Here Po,X.1 = T 80,
[ p Po log
Po
/ pp<n Po log Po + log Zm poen
/
Po S cnn.
n/
for some constant d' > 0. This follows since on (po < n), Po log(po/ 5..
(ii Region X Here Pa.X.1 = Po/Zuy and,
f Po log
Po 1,)) =
f Po log En S E"TT, for some constant c" >
(33)
Po DolZrn
Now collecting the bounds from (i) (ili) and absorbing constants yields
DRi(Po p...)) s Cin + C271-
Proof of Theorem 3- By Lemma 4.1 of (Zevv & Meir 1997 for any E > 0 there exists an ê-component Gaussian mixture hg A such that
DRIlpe,x.n ha- < O of
(34)
and, the compact set X, the mixture satisfies uniform bounds n S (o) he (o)))(x) I M, e X, for some finite M > 0. Now
(RI(polbh) DRu(Pol1) + (RL(f())) + (Eps EEy) log h
with f Pa.x. and h ho (35)
13

---

Published in Transactions on Machine Learning Research (01/2026)
By Lemma we have, for constants C2 0, (R, 1, and 1
DKI(Polleo..a) v Cin+ C2nr C2 7) log(1/n) (Using AM-GM inequality) = =O + (o0)))))((\/1)
(36)
The last equality follows since 7 = (() by Lemma
2 Now for the last term in 35 we have,
/ (po(x) f(x)) log f(x)
(Epp E,) log h.
de,
h(x)
(Bps E,) log S
I llpo fll1 sup log ((x
h(x)
On X, we have n/Z,- S f(x). h(x) S M for Zn E [1, 1 + 17) 1], so
MZn
sup
log
(ix) h(w))
S log
= ((log(1/n))
=EX
1
and outside X the contribution can be absorbed into the constants since the tail mass 7 is O(r). Lemma implies
llpr - S11 S CTT + x = 0 - O(n).
so,
(Ep. - Br) log f
<
(O(1) O(a)) log (1/n),
(37)
Finally, combining the three bounds 34
36
and
37
in
35
obtain
DKL (Po (1a
< (O(n) + O(o)) + o 172)
+
o
(38)
which is the bound claimed in Theorem
Theorem shows that the mollified stationary distribution v/ vb o(0,,21) admits finite-ixxuree
Gaussian approximation with controlled KL error given by,
D (0) (1((=;0,>*)) s O(n + σ) log(1/n) + O(E/n²) + 0(1(G28)).
where v is the clipping level and is the number of mixture components. The first term captures the error introduced by mollification and clipping. Since (1) σr) log(1/n) -> as 17. + 0, this contribution can be made arbitrarily small by choosing y) and sufficiently small. For any fixed 17 0, the remaining terms decrease monotonically l increases and the approximation error e is reduced (e.g., by refining the mixture class), and the empirical curves in Figures. 2a and 2b illustrate this decay. This guarantees that h(=;0","") converges to v/ (o) in KL, and hence in total variation by Pinsker's inequality. The resulting mixture h(=;0,,* therefore provides a numerically stable surrogate for vs and is required for the distributiion-corccttioo step of our algorithm.
We therefore compute the optimal surrogate by directly minimizing the KL divergence to the mollified target (()) Specifically, the mixture paramcters are obtained by solving the optimization problem:
aeer XEAr DKa(K) ((:;6,))) 2)) arg max x²) arg min
0 X [log h(x; 0)).
The equivalence to maximizing the expected log- likelihood of the mixture under the mollified law follows from the definition of KL divergence. The objective differentiable and also well-posed bocause v(() is absolutely continuous with respect to the Gaussian mixture model, ensuring the KL divergence finite and the optimization landscape regular. The convolution structure further allows to rewrite the expectation
14

---

Published in Transactions on Machine Learning Research (01/2026)
True stoady statt dist Busion us
Goussian macure (00,0.00
KL divergence vs for different bandwidths
0334
17.5
1.0
0.7
8415
15.0
o- 0.4
0= 0.2
Dx.Val he,a)
12.5
10.0
Gauasian midiuw 222 itm B.d.
Gauasien micuew 488 am
664
602
0.00
10
30 Number components !
20
40
50
(a) True stcady-state distribution My lcft-left) and its Gausslanmixxtrre surrogates (f, {(9,0.7) (25, 0.4), (49, 0.2)3. As l increases and creases, the mixture becomes close vb.
(b) KL-divergence DRI(Ps ll hxxx) versus component count for {1.0, 0.7. 0.2. Curves decay monotonically, corroborating the & and dependence in Theorem
Figure 2: (a) Moussian-mixlum surrogates approaching the mollified stationary distribution; (b) KL decay with increasing component count.
in terms of samples from the original Markov chain. If we draw St vs and ~ N(0,021) independently and let X S; + Et, then x2 r//,) and E x [log h(x: A)] Ee e log h(s EG; A)], producing unbiased stochastic gradients. Now let.
F(O, A) = B.g.o (oi (xx;0oxxxxx;0111
(39)
Then, the dominated convergence theorem yields VF A) = E E Plogh(x; A),, so the gradient coincides with the expected score of the mixture.
We now leverage the above results maintain a running estimate of the approximation of is through online maximum likelihood estimation, where the parameters (0, X) are updated via. incremental projected stochastio-gradien ascent with Rulpert averaging (Polyak 1990 Ruppert 1988). At iteration the noisy gradient V log h(:; Oz, Ae) replaces V (F Az). The mixture parameters evolve ve on possibly different step-size sequence (at) than the TD parameters (Bx), allowing flexible calibration of the density model relative to the corrected value estimator. This two-timescale structure couples the density estimation with the off-policy correction step and supports convergence of the overall algorithm.
0++1
((2)
ars Vo, x log h (Sc 0,, (,,
Ae+1
(40)
θ 0
0x
1
10+1 det1
0₂ N
+
t+1
where 5,41 = St41 + E the state sequence (St)c20 is generated by the behaviour policy as a Markov chain on S, and Et+1 No0,, I² an i.i.d. Gaussian perturbation independent of sy. Under standard ergodicity assumptions, the chain admits unique stationary distribution v, and the empirical law of s, converges to up: in the mcan-field and ODE analysis therefore interpret the expectation of the update in (40) with respect to the mollified stationary law v)")) Further, Il e is the projection operator, which projects 0, onto the constraint set O and X, onto the probability simplex A This ensures itcrates [0,, A,] stay feasible. In the interior of e A it acts as the identity, and near the boundary, it projects orthogonally onto the boundary. Here ae (0, 1) is the siep-size parameter, fixed apririi. Dalpek-Ruppert averaging is employed to enhance
15

---

Published in Transactions on Machine Learning Research (01/2026)
stability by maintaining running averages 8e, of the stochastic iterates. This averaging scheme reduces the effects of noise in gradient estimates and provides more robust parameter estimates for the distribution correction step.
Remark 2. By leveraging the properties of the NEF, one can obtain a closed form expression for v loo h(:00, X) as follows:
0
v log h(; (T(x) (K))))) (s)
0
(s;
0
90, (s)
O,2 log logh(s;
h1s; O)
In our algorithm, we use a timetite stochastic approximation framework. The stochastic gradient ascent for tracking the steady-state distribution and the TD recursion for the off-policy solution are updated faster limescale, while the PR averaging step is updated on a slower one. Specifically, the step-sizes for the gradient ascent (az) and TD recursion (3t) are orders of magnitude larger than the PR-averaging step. This means that while the faster updates capture rapid changes, the slower, smaller step-size of the averaging step smooths out the fluctuations, stabilizing the learning process and reducing noise. This timescale relationship is formally defined as follows:
ar 3. E (0, 1),
Σas= 120
3 0c,
Ela + oc, an Br R2( t t+
1
(41)
12:
$20
Further, we modify the TD recursion to correct the stcady-state bias by incorporating the stcady-state distribution correction factor (G) as follows:
T(a (s)) and Se q(st)
Xe+1 xx + SepiCe (ret1 + 1 o x1) de, where de ((s:), Pr: (=)a)))))
(S O2, Ax)
8e TD error
442)
Intuitively, Pr reweights the TD error & by how likely the chosen action is under the target us. behaviour policy, while &a reweights by how likely the state 8, is under the behaviour's steady-state distribution, w.r.t. the desired design distribution q. By introducing the correction factor St, re-weight updates to emphasize states in accordance with a predefined design distribution q. In the idcal case, we set g(s) u* (s) (the target policy's 's true stationary distribution), but even if VT is unknown, we can choose ((s) to be a reasonable proxy. It is a predefined, domain-specific distribution or heuristic approximation, carefully handcrafted to suit the problem context. For instance, in risk-aware or emphasize certain critical regions of the state space, while healthcare, it could overweight underrepresented patient conditions ensure equitable learning. This correction adjusts state visitation frequencies, ensuring that states infrequently visited by the behaviour policy receive appropriate weight during learning. The pscudocode of our approach is given in Algorithm [
applications, may
Assumption 4. Geometric mixing (spectral gap). There erist constants M > 0 and P E (0,1) such that || - vollry I M s t 0.
Assumption 5. Parameter Space Regularity.
The parameter space θ is compact with smooth boundary.
Assumption 6. Uniformly bounded score function. There constant G 00 such that (Vlogh(s 0, [. G of VX A E.
we note that the parameters = (0, Remark 3. For the miature NEF model h(s; 0,A) E,
le in a compact set (because θ is compact and X lies in the simplex, which is also compact). Then, for each s, h(s;v) is a continuous function of (as finite sum of products of continuous functions) and hence attains minimum and marimum over the compact parameter space. Again, because go 0 and X 2 O with Σ);- = 1, we have hs)s) 2 min; ga, (s) 2 1) >> O and (x) S max max (x) S M. Moreover, the same bounds hold uniformly in s because there are finätely many states. Thus, for the mixture model, we also have:
0 1 ((8 S M < X0,
VUE A VsES.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm 1: Off-pooicy TD with linear function approximation and distributional correction
Function Off-ID-SSS -D--SSDCOTTT TT)
for each transition (st, ag, 1, St41) do Calibrate parameters as follows:
[0+1 Ae+1]
Il
1e A
a V log hogtts1411 de, 2.))
where $141 S141 + El+1, and EL+1 ~ o0
0241
Oz
1
[r++]]
0c
Ae+1
Ad
++1
[ut11
[ee
X++1 = Xt + BepeCe (re+1 + 70(+1 Xt - De Xx chr
πl (a))st))
g(s:)
where pr
and Gi h(s;; O,, x,)
To To(a, (st)
This then leads to the boundedness of the score function as previously explained.
To establish the convergence properties of Algorithm [ we analyze the stochastic updates of the dis- tribution parameters 0 and λς. Lot Vt A., h(: eq) h(= 02, Ae), U @' A and F, (0%, Åe, 2g Åt, Sk; ax, Xt: Ck. < < t) be the natural filtration generated by all variables up to time t. Then the update recursion [0,, Xs] can be decomposed into a deterministic drift, a martingale noise, and bias follows:
Ut+1 = -ly (U: + ar V log ((5++15 vz))
= -Ily (vr + a (VF(w)) + M + b;))
where M,++ = V log // S v2) - E (V log (5 F.), and 62 = E EV log44(4411 vt) - VF(v)).
(43)
First we establish a fundamental result on the bias term which shows that the bias term is geometrically decaying and therefore summable.
Lemma Let Assumptions and hold. Then the bias term by satisfies ||6||| s GM p 2 and
| < co.
1=
Proof. By conditioning first on 8:41 and then on the Gaussian perturbation, we get
V log h($ |F] =
Σ s'ES
Pm (s = gu.
(44)
where we define the mollified score (s'))) Ee-, V log his' +c; Under Assumption 6 the (un-
perturbed) score is uniformly bounded, V/ log h(x; v) | S G for all and hence [s))l s G for all s' e and all u. The gradient of the mollified objective F(U:) is
F(())) = Egom. e~A((0.021) log (sgghss+eii {= Σ vo(s') go. (s'). FSS
(45)
Subtracting
from 44 and let mo(s) PT, (s: s), we get,
Σ (u((s') - vo(s')) go. (s)).
(46)
s'E8
Using the earlier uniform bound 11g,, (s')ll < G and the identity 1|me - -ubl11 = 21/ VbllTv, we obtain
|abaa| Gl| voll1 2G 1 1b TV-
17

---

Published in Transactions on Machine Learning Research (01/2026)
From Assumption 4
(geometric mixing of the behaviour chain), we obtain
||bس|| - 2GMp',
where M and p 1.
Further,
Σ t=0
au 0r116
S 2GM
app'.
t=0
Since the step-sizes satisfy a, - 0. Σ, 0 0c, and Σ, o 0o (from 41
the weighted geometric series
E, app' converges. Thus Σ, a. 00.
The following theorem establishes that the sequence A converges to Karush-Kuhn Tucker (KKT) optimality conditions where the gradient aligns with the normal cone of Of x A This guaranlees the learned mixture distribution h(; 0, converges to stationary point of the KL-divergence minimization problem.
Theorem (Convergence Distribution Approximation). Let the step-size {au} satisfy 11.. Let Assump- tions hold. Then the sequence {[0, }] converges almost surely to the set of KKT points:
(v = 10, E A: -VF(u) E Nu(u)},
where (v) denotes the normal cone to Theta x A at v, defined as:
Nu(v) {d pdim(ef)+ (d,u v) s s0, VuEU.
Proof. Let 9t = VF(()) + MH++ + b.- Then,
U++1 = le - a(gi),
= + ( + IT (va + 019:) (
Ut ta [T(( + No (vr + o1gi) lr(())
= 1 + (gu( + o((a)))
(47)
The last equality follows since
By (vr + Egt) U II (o))
lim 80
(48)
To(a)
where
(v) = {u E Rdim(0')++ v ru E U for some T > >0}.
In the interior of U. (v:) = +z (unconstrained) and near the boundary of U, Tv( = u e R4 u points into U}. Thus, (g is the directional derivative of the projection operator Hg point Vt in the direction gt, which is equivalent the projection of 9t onto the langent. cone of U at v. [Rockafellar 2015). Intuitively, it captures the "feasible component' of g. that aligns with the constraints of U.
For the noise My B[M., Fi] = (by definition). Further, using the triangle inequality,
MM |ll
Vlog h(5 v.)) EVVlog h(5 v) Fill
G G 2G a.s.
18

---

Published in Transactions on Machine Learning Research (01/2026)
By squaring and taking conditional expectation, get B|||M[-11 1122 [i] < (2G)2 4G². Thus {M] is
iquare-integrable martingale-difference sequence. Now, consider S, = Ei 0 Note that
Σ
E [|| - [. Σ af ||M 4G Σ af < 00.
(49)
120
By martingale convergence theorem, it follows that S, converges, i.c., Et= a, 1411141 < 00 a.s.
Now rearranging we gel.
- Ut = u; + (Fo((((() + +ov(9)) - ( (V(V(())) +o(a.))
(50)
&e
Using the non-expansive property of Tv, we have
|k&lll = [T - (V((V(()))
I 19. VF((t)]] = ||M]+ + 1
|M+++ ||],|||
(51)
Hence,
Calllell & Σ MMall + Σ as |b;l < 00 a.s.
(52)
Therefore by [Borkar 2008). follows that {ur} asymptotically tracks the ODE
= (VF((v))
(53)
However, because F' is smooth and the constraint set U is convex, the above differential equation is well- defined and corresponds to the projected gradient ascent. By the theory of stochastic approxination (see [Borkar 2008)), ), the sequence {uz} converges to (passibly sample path dependent) internally chain transitive invariant set of the above ODE. Since is C
d ((((t) (VF(u(t)), b(t)) (VF(v), [o(V)()))) di
Apply Moreau's decompasition to obtain VF()) = (T(()) [F(e)] + TNo( [VF(v)] and (Ilro()) [F(o)),[lNdte [VF())]) = O. Then,
(VF(C).Tc(F(()))
(Ilr [F(()) + NNCO [VF()) (((F)))
(TT(VFFU))
= =|\To(FF())] 2 0.
(54)
Hence
& F(v(t) [[c[VF((())]]] 2
(55)
with equality iff [u[VF(u(t)] 0. Therefore, the invariant set of the above ODE is the stationary (oquilibrium) set:
[ = 0) = (VEU: e No(v)),
which are the Karush-Kuih Kuhn-Tucke (KKT) points. The last equality follows again by Morcau's decomposition of (F(v).
Having established the almost sure convergence of the distribution parameters {vc} to ot in Theorem analyze the temporal difference learning dynamics given by 42 Prior to this, observe that the
19

---

Published in Transactions on Machine Learning Research (01/2026)
unilateral timescale separation between the faster distribution estimation updates (v, = [0, ), and slower Polyak-Ruppert averaging (6z [0t, ) ensures that 0z + o² asymptotically. This justifies replacing the time-varying = g(s))/h(ss)) (,) in the TD update (42 with its steady-state counterpart Ç = q(s))//(s;; v*).
The substitution decouples the distribution approximation error from the value eslimation error, permitling the simplified TD recursion (See Chapter 6 of (Borkar 2008)) ). We need the following assumption on the design distribution qF
Assumption 7. The design distribution q strictly positive over all states: q(s) > 0 for all E S.
Now, we rewrite x, update as follows (we let gf Pt S2 (re+1 + 10 of z) oe and h² h²)))) h(; ());
<(aa(s:)) and C h (s;)
((st)
X++1 = x + S.Pt57 (re+1 + 1 X, of xe) where Pr To(a, 8,)
= xx + 3r (b + G(x)) + M[+1).
+
(56)
Here,
G(z) = EgggT] = 18 C7 (re+1 + -
g(s)
Eg (s)
E Pe [T++1 + 1 0f x) s]
(())
F [h)(ss)
(R.(s) + +- (p) (Fx) (bss)
= -E,,, Ep. Eg (Rx + (7 P. - 1)
+ TE, Spi E. (7 Pr 1) x + + E Sp E R
(57)
Ac
Also,
MI 9: - E[gee [i], and by = E[g: [i]
Mxx),,
where 9: Pc Pe (re+1 + + - - x,xx)
(58)
Further, note that since we have finite state and action spaces |re| S Roos [[{ < Poo, and 0 Pr Poo: 0 << Se Soo.
We first bound the TD error as follows:
5g1 = |ret1 + 7 - 0, x₁
|r+1| y10 x10 10, x,
Rsx + 10 + 4 ||
Rsc + 7) |x/||
Then the update term 9i satisfics
19e = / || S Poo (Rox + (1 200 20 SC C2|x
where Ci = Proo Rox oo; and C2 = Poo + D2. Now,
|M || = |19t - Blgg Filll
< |1ael| + ||E[ge FF]l $ 2sup [9
< 2(C1 C21[xx|1) < Č1 ||x:
Further, using ||a - b112 < 2||a||2 + 2| we get,
E [[IM 2
Fi] E
||
Elge F:1ll Fi
< 2 E [[|9: 12 Fi]
+ 2 ||x F:11|?
< 4E 4 [1
[F]
S 4C1 + 12)
< SCp + 8C3
(59)
20

---

Published in Transactions on Machine Learning Research (01/2026)
Now we write by Eige [i] G(x:) Elge (s, xx] Ev, l9el-
Thus, ||b1| || = ||18[9 [8,, xi! [aelll
- Pr (st, s) 1s(s) || st. xI]l
.ES
I sup |B[
8, x.]l I| (st, :) 1b
= 2 sup |[[|
s, s,xI'| || (St. 1olTTV
S 2M (C1 + C2llx - E2p' (1+ |1x+])).
(60)
To establish convergence of the sequence {xt); we must first ensure the itcrates remain stochastically bounded. While classical stochastic approximation theory (Borkar) 2008) oflen assumes almost sure bound- edness, prove the following weaker but sufficient condition for our setting.
Lemma 6. The iterates x2 satisfies sup, EE [||x;|]]] 00.
Proof. Note G satisfes the drift inequality
x G() S -c|1x12+ + E
(61)
where c A2min(-AA (-Ao) and d =
Using E[M[++1 Fi] = and expanding the square,
[ e++ FF]] = ||x ||2 + (G(x)) + b.) + az ( + + + 112 |Fil).
Apply 60
59
61 and Young's inequality 2x, by s c||x c- 11 and the bound ||G(x;)]12 s
2||A.11 1 X₁
2|| to obtain
[[[[+1] S (1 - (2c - c)ac + 1.2 a² [[[ + 2d + To + (0 an + 20²) 8M ( + +f). (62)
where L2 = 4// + 8C2, Lo = 41 + 8C7.
Further, by rearranging, we get,
E[Ve+1] S (1-c5) cBe + Lyff + SM (c + 2
E[V. + 2d + Lo of + SM²C C2 (c-1 + 2
=G.
(3)
Since 2²1 + 0 geometrically and 3r - O, the perturbation terms Gr, e, vanish; for all large one can ensure G. < 53, leading to the following stochastic approximation form
E[V+1] s (1 - (3)) EVV. + 2d + C
(64)
Now using mathematical induction, we will show that sup, E[Vi] < xo. For t = 0, [V]] = = is finite
since Xo is initialized with finite variance (base case). Now assume E[V; < K for some constant K and all
< T, where
4d + sup supd K = max [[||x|||
2
120
Then,
B[V++1] < (1 11 (1-(8r
K + +d3d + c33
-- K 2d + c
< K + 3r
(65)
21

---

Published in Transactions on Machine Learning Research (01/2026)
4d 2
Since K supp Be, we have 120
(4d 4d
kK22d= +22
K
sup c' B1 2d
2
120
supssppi 31 S - BT < 0. 120
Thus from
65
we get
B[VT+1] S K- c'a + caj S K.
By induction, [[]] & K for all but a finite number of t.
To establish the convergence of X we must show that the bias and noise terms are manageable. Specifically, the next lemma establishes that the series formed by the weighted bias and martingale noise terms converge almost surely.
Lemma 7. For the martingale noise My and the bias bi, we have
P Σ B.Mf++1 < oo. 3ibg < 00 =1.
Proof. From 59 and using tower property, we have
M[[ 1 FF S 8C + +C |x,
E|| M Il & + [
(66)
Hence, M/+1 is square integrable Now, by the convergence theorem for square-integrable martingale (for vector-valued martingales), it is enough to show that
1[1 [A] < x 0.8.
Thus it is enough to show that
E
Σ BIIAMAI Fi)
< <0.
Therefore, by monotone convergence theorem, we get
Σ B| 212 Fi]
Σ B Fill
Σ 83 (8C 80 || 11
Σ 8 SC? 8C sup E|| 00.
The last incquality follows from Lemma 6 and E, 83 < oc. This implies that P (E, (E, <00) Now for by, it follows from 60
E
Be 11
2M
Σ
C2 12)
2M C2 sup E X
22

---

Published in Transactions on Machine Learning Research (01/2026)
The last inequality follows again from Lemma 6 3. 0 and p E (0, 1). Hence, P Sebl+1 < 00) 1. ]
Having established the stochastic boundedness of the itcrates X and the almost sure summability of the martingale noise Σ 0,M441 and bias lerms E, Orp we now prove almost sure convergence of the sequence {x.}.
Theorem 5 (Convergence of the TD Iterates). Let Assumptions 7 hold. Also, assume that the matrix A. Φ E, I) is Hurwitz (oll eigenvalues have strictly negative real parts) and diagonalizable. Then the sequence {x+} converges almost surely to the unique solution x' =XYD satisfying:
Φ (, - Tx -0 = E Roy
Proof. Rearranging recursion 56 of X, as follows:
x (x + 3,Ac(xi x¹) aar + Beb +M++1).
But note that Axx + E = 0 by definition of x"x So,
X++1 x = (x: - x*) + BxA -xx
x*) ) + Be + M,+1)
Let C X2 x". Then,
6t+1 = (1 B.AoJe: BeAAeee +
Ben. where Nhe bī + MT+1
We know that E. Bellvell a.s. by Lemma (since both by and Mf summable in absolute value a.s.). Now, because Ag is negative definite, the matrix I + B,A, has eigenvalues in (0, for small 8,. After unraveling the above recursion, we obtain
i II (1 BANA)
I Bade)) yo + Σ BATE A=
et+1
k=0
j=k+1
Let
t-1
k(t,k) = II (T+ BrAc) for k. Q(kkk) = I.
(67)
j=k
Then,
c+1 = 1,0)e + Σ 3xQ(++ 1, k 1)7)-
(68)
R=0
Now, since Ag is negative definite, let Amin > be such that the real parts of the eigenvalues of A are less than or equal to -Amin- Then, there exists a constant C and such that,
[1((1,)) k)]l S Cexp
-3
(69)
j=k
Since Ae is diagonalizable, let A, PDP-1 where diag( (i )d) diagonal. Then,
[-1 Q(t,k) II (1 + 3yAc) =
1-1
[ [[[++3DD]] p-t
j=k
j==
P
D2
p-1
b=k
23

---

Published in Transactions on Machine Learning Research (01/2026)
where Dj - 3,D diag(1 + azA1 ....,........................... 1 ByAa). The norm satisfies the following:
1Q & |PI|| |P
Dj
(70)
k
The diagonal matrix norm is given by:
t-1 Dj
[-1 II 1
max Isid
+ 3,
Now we establish uniform bound for each eigenvalue product IT;= Bz)a). For any 0, there exists 3o > 0 such that for 0 S 33 < So:
11 Bydel S Ee Ro(Ai)+
(71)
This follows from the logarithm expansion:
log(1 + B5Aa) = ByA; (Bg2i)2
(e())) + i3, (A:) + 0(53)
the real part is a, Re(A.) + O(a3). For sufficiently small aj, have: Re (log(1 (3A;)) s 83 Re():) + cB3
Thus |1 + 3yAc = [Re(log(+2]AA)) < 3, Re(
Set e Amin/2 2 where Amin = min: Re(A,)). Since Re(A:) S -Amin
|1 ByAs S e Re(Ar)+(55mmndd e -0 Amin +BzAmin/2 c- Amiii/
when 8g Bo. Since 8j 0, there exists Ko N such that B5 8o for all j 2 Ko-
Case 1: k 2 Ko
For all j 2 k 2 Ko, we have Bg S So, so:
t-1
Amin t-1
II (1 3.Ab)
exx
Σ 33
1=
j=k
Case 2: 1 < Ko
Split the product at Ko:
1-1 I 3,A,)= =
Kg
[-1 II j=Ko
II j=k
(1+ 332e)
j=
(x*)
Term (+) is finite product (since K2 is fixed). Using 11 + 3yA.1
S 11 1e
Kg-
Kp-1
1(+) s II (1+ (+11)11))))))))))))))){1) S exp |
Σ
C.((k)
j=h
j=b
where C,(k) exp (1N Ko-1 3,) is bounded (as By and fixed Ko). Term (**) bounded by Case
Amnin
[(**)| S exp
Bj
2
j= Ko
Amin
Amin
exp
B
exp
B
2
24

---

Published in Transactions on Machine Learning Research (01/2026)
Combining both terms:
1-1
Amin
Ko 1
Amin
[-1
II (1 + 3,2,)
S (kkk exp
Bj
exp
Bj
j=
j=k
==
1-1
Amin Σ
= Ck) exp
83
j=h
where (k) Ci(k) exp
سسسn jz= 83).
Since k < Ko and there are only finitely many such k we define the following
C' max
max
C.,)) 1
00
Isid
OSk<Ko
For 2 Ko, we have (k) 1. Thus, for all i, k. and k:
Amin t-1
II (1 3220)
s exee
Σ 33
j=k
Therefore,
Amin
II Dj
max II (1 BBA,)
S CC'eep
Bg
2
i=k
Substituting into 70
(Q(t, Ri]l < |||| | C' cxp
Amin
Σ
1-1
33
2
z=k
Set C = ||||| 1||-111 O and B = Amin/ 1 to obtain:
1-1
[[Q(t,A]] < C Cexp -3
_
Bg
j=k
for all t 2 0, with C, > independent of t and k.
Therefore from 68
1 -B
|es+1|| |l S Ceep
-.,,, 3i |col| + Σ 3e 1Q m)
(72)
1=0
k=
The first term goes to zero as t + because Σ₁= Bj + co. For the second term, note that
Σ B 1)ll mKl S Σ
Be exp
-9
Bi
|| ll-
(73)
j=k+1
By the summability of B& |mll and the exponential decay, this term goes to zero. Indeed, for any fixed k the term goes to zero as f - do. Moreover, the tail the series En 8x ||Ta|| is small. Therefore, by the Toeplitz lemma or direct estimation, the entire sum goes to zero.
Thus, e - 0 a.s., i.e., X, x
25

---

Published in Transactions on Machine Learning Research (01/2026)
A natural question is whether our correction mechanism guarantee that the residual bias stays pro- portional to the unavoidable approximation error. The next theorem answers this affirmatively, showing that the corrected TD fixed point is never worse than a constant-factor multiple of the best value function representable by the chosen features.
First, we define the total error as:
- 2x V
(1xx w²) + ($w" VV>) 4u + 6,
(74)
where
w" = arg minl I Villa is the best approximation under g-norm
u ze TD w is the difference between the TD solution and the best approximation
8 vw V is the approximation error Lemma 8 (Orthogonality Condition for TD Fixed Point). The TD fixed point sotisfies the orthogonality condition:
+ ". >Pp)e e== =0
775)
Proof.
The TD fixed point satisfies:
Substituting the Rr = =1 -I- (from Bellman equation) into the TD fixed point equation:
T - E -P= = + E,, -
Rearranging all terms to one side:
T' 7Pa) V²) =0 = Φ S >P>)c= =
To rigorously validale the effectiveness of our distributional correction mechanism, we bound the approxi- mation error relative to the fundamental limit imposed by the expressivity of the features and the target state weighting. The following result provides worst-case guarantee that our method does not amplify unavodaable approximation errors and quantifies how design choices (e.g., the target distribution q, feature selection, and mixture model complexity) influence performance.
Theorem (Error Bound for Off-Policy TD with Steady-State Bias Correction). Let A. be Hurwitz and diagonalizable and Assumplions hold. Then, the error of the off off-policy TD solution with steady-state bias correction satisfies:
1 Vx Valles S C min fw Vz
where:
||P|| ||P||-11--11|
max max (b()) K. max g (5) (1+ + max "b(s)
(a(Ac)l
g(s)
Σ, ((s) and a(Ae) = max (A.A.(AA)) < 0 being the spectral abscissa of Ac-
with Kg max
g(s')
26

---

Published in Transactions on Machine Learning Research (01/2026)
Proof. By applying triangle inequality
74 we get
| 11e 1 + +1111
(76)
From Lemma we have
(t Em -1) =0
(77)
Substituting e e Du + 8:
+ Eug 11- - 1) - -E, 1)6
Aou -b.
where by = = E,, - 1)6 Since Ae Hurwitz and diagonalizable, Ae is invertible and can be written as A PDP-1, where D diag(At,...,AM))) Ax) with Re(As) < 0 for all i. Therefore,
u == -A by = -PD- bb
(78)
By taking norms on either side, we get
|||| $ || |lbs 11 S
||l||
||-11|| 11 110 11 ||68||
(79)
Since D is diagonal with entries Ay:
1 X ming A
| max
Let a(Ac) = max O. For any eigenvalue A, a + bei, have |1d Therefore,
Va + b2
|a,|
[Re(A))
| (a(Ae)
(80)
So, from
79
we get
|||| - 11P
[a(A>>] ))
||u||
||bs|l
(81)
(a<AAe)
Now we bound [|bsl| = inequality:
- T)8l1- Using the g-weighted inner product and Cauchy-Schware
1v 6sl = 1((s///"))b (xPx - 1)0)al
- (1(16/11))00l 11 - 1)01l
(82)
We first, bound (1r//h" h/ Bull::
Pb(8))
[1 (wm/h"" ( 2000
Eg(s)
(s))
(Dv(s))²
T+*(s)
< max (b(s) 2
max (s)) . |
(+)
< k22 max qss) maxx (2) ||a||2
Therefore,
1(mz/h")))ll < K
max q(s) o Taax (i)
||0|||
(83)
Next, we bound [(yp, - 1)llly Let mls') E, Px
which is the next-state distribution under one can obtain the following:
policy a when starting from distribution q. Similar to Lemma i
s max 14(8)
|| Pr /I = {{)))()(()) < Σ ((s)) Σ
-1551
((s')
Kg
27

---

Published in Transactions on Machine Learning Research (01/2026)
Then, |P I Vag. Therefore,
11<>, - 1)0la S 7 + 11811
S Sinigg 1161l +1161a = (1 (1+77 11811
Combining these results:
|bsl|| s K
max ((8) max ( (1+7(177 7//ki)) 1101|
(84)
Now, we bound the approximation error under vb:
| = Σ (b())((s) < max pb(5)
((5) ) Σ [(())(())² (maxx
[max vb(()
g(8) 1|01
Therefore,
|8110 S max vb(s)
|001
(85)
(ss)
Finally, combine all components in 74
|ellvs - 1u + -10011e
I Pl| |PI 11P-11)
)) max mmx vs(s)
vo(s)
a(Aa)
max ((s) mmax (2)2 (1 (vg)))
-1811a +
max
g(s)
-166'l
Since
||ō||, min.. | Vz and by the definition
74 of we obtain the claim:
| TD Vallo, S C min ||o llg:
(86)
where
||P|] |IP|| [1P))-1P-11)) a(Ao)
max g()) ( (1 (1kg + max vb(s)
C
max
((s)
The above theorem demonstrates that the error of our corrected solution is proportional to the minimal approximation error under the target distribution g. scaled by factors capturing policy misalignment, fcature conditioning, and steady-state estimation accuracy. This establishes that our algorithm achieves near-optimal performance within the constraints of the representation, while explicitly quantifying the cost of distribution shift correction. The bound further elucidates the trade-offss between policy similarity, distribution estima- tion quality, and feature design. This bound further provides several relevant insights, which are in order:
1. Fundamental Error Relationshép: The error in the TD solution is proportional to the best possible ap- proximation error, establishing that the algorithm achieves the best possible performance within the function approximation class.
(x(s) k(s) quantifies the impact of steady-state distri- bution estimation error. When K z 1 (accurate estimation), the bound tightens, validating the steady-state bias correction approach.
2. Steady-State Estimation Quality: The term K = max,
3. Policy Alignment: The term (1 + kug) with Kg = max,
q(5r) measures policy dissimilarity. Smaller a (more similar policies) leads to tighter bounds, explaining why off-policy learning becomes challenging with dissimilar policies.
mls')
4. Feature Represenlation: The term Tmax m shows that well-conditioned feature representations (smaller
m lead to better error bounds.
5. Distributional Factors: The terms / /max, (s), / max, 9(8), and v mmax, g(s)
2y(s) capture how state distri-
bution properties affect performance.
28

---

Published in Transactions on Machine Learning Research (01/2026)
While Theorem provides error bound under the behaviour policy's stationary distribution Pa, in many practical scenarios we are ultimately interested in the prediction accuracy under the design distribution g. The following corollary establishes that our steady-state bias correction method also provides strong guarantecs in the design distribution norm.
Corollary 2 (Error Bound in g-Norm). Under the same assumptions as Theorem the error of the corrected off-policy TD solution under the design distribution satisfies:
g(s) C,
1 TD V=ll, S Cg min fww Vallg. where Cg= = max aE vs)))
and is the constunt from Theorem
Proof. Since the state space S is finite and vs(s) > 0 for all s E S (by Assumption 1. msx 4(a) is finite. Hence,
vs(a)
|xx
TD
Vrlli =
Σ
(xx TD - (+)
#ES
((()
Σ (5)
(fx TD -s))
vb(s))
#ES
max ((6)
(s)
(fxx (=)
#ES vo(s)
#ES
((s) max #ES (5)
|
TD Vz 11
Taking square roots and applying Theorem get,
((s) SES (s(s)
g(s)
| TD
Vvl1 S
V
max
[x TD
Villus S
max SES 2)(3)
C
min w
pww Vallg
Remark 4. The constant Cg reveals an important trade-off: while our correction mechanism aims to align the solution with the target distribution q, the final error bound depends on the marimam density ratio between and Vs. This highlights the fundamental importance of coverage: if the behaviour policy rarely visils states that are imporlant under q (i.e., vp(s) < q(8) for some 8), then R becomes large and the bound degrades. This aligns with intuition and provides theoretical justification for the empirical observation that good behaviour policies should have adequate coverage of the target distribution's support.
Remark 5. When q Vx (the ideal case where we know the target policy's 'sicy's stationary distribution), Corol- lary provides bound on the evvor relative the true evaluation metric of interest. The constant C2 then depends on the distribution mismatch coefficient max, M=(s) which quantifies how well the behaviour policy covers the target policy's 's state visitation pattern.
vo(e)'
To complete the convergence analysis, we now provide a sufficient condition for the matrix Ao to be Hurwitz, which ensures the asymptotic stability of the algorithm.
Theorem 7 (Hurwitz Condition). Ar Φ EE, -TT is Hurwilz (all eigenvalues have striclly Py(s") with py(s") = -. (s'))( and negalive real parls) if and only if Kyngn² < 1, where Kig maxx' a(ड))
K2- max, Eng (s)
(3))
Proof. Consider the quadratic form w A.w for any + 0:
w A- w E. -J/o
29

---

Published in Transactions on Machine Learning Research (01/2026)
Let dw. By Assumption
2 + 0 since rank(()= = k. Then:
w Now w u E,, - , (7 Px T)u = y E, _ E,.
Let Q1 ,,, and Q2 u E. E. First, observe that Q2 > 0 since E - Fy are all positive definite diagonal matrices (by Assumption and the fact that ht is valid distribution estimate).
For Q1, apply the Sauchy-Schward inequality:
Q11 S E, || E,,,,, where |xx,,,,, Ey
Now, we need to bound
Px _
Pr 2
=u (Px) Ey
Eug -
ΣΣΣ
u (8) (1)1)))))))))))) 10
8"
(+) 5s(
Σ=
Σ
(=(())))((s)
Note that Kg- max, Ex, (.) 0(3)
is bounded since vs and h" positive distributions finite state
space. Then,
|Pau (P, STLE.
Kg Σ (())(())
Σ PP((d(o)u(s))
K Σ a(())u(())
(s(s))))(()
(by Jensen's inequality)
kg/
Σ ((s')u(s))2
Σ Ea(o") ((()) Σ (s))s)))(()
=Kyllulla Eu(s)² Σ ((s))Pa(s'(s))
-Killullo
Σ ((s))pp(s))
where Kig = max Mg(s')
< KgyHigllll?
((s')
Thus, | S., 2,13, S (Kgtillullyy Now, since ||u|| B,, =, v Kg
||llg we have,
10 TV Kotig |u ST
Therefore:
w N.w < Kggg 1)llu| 1| E,,=,
When Kn < 1, we have TV // Ka < 1, and thus w -TA.w w < for all w + 0. This proves that Ag is negative definite, and therefore all its eigenvalues have strictly negative real parts (Hurwitz).
30

---

Published in Transactions on Machine Learning Research (01/2026)
In Algorithm the mixture weights X must satisfy probability simplex constraints (A, 0, > 1) after each gradient update. To enforce this, we employ an efficient projection method that provides an optimal O( log f) Euclidean projection onto the simplex A (Wang & Carreira-Perpinéu 2013). The method sorts the components of X, determines an optimal threshold, and redistributes mass, providing the closest valid point the simplex A while preserving sparsity patterns when possible which altributes to its O log E) complexity. Specifically, soeks the solution to the following optimization problem:
1 min د v13,, subject to
> =1 A; 420. 0.
1=1
Using a Lagrange multiplier 7 for the equality constraint Eim A = 1, we define the Lagrangian:
1 2 Σ (2
- Σ A -1)
1
C(),))
-7
i=
Now solving for X by taking the derivative w.r.t. Ap:
OC = A - 1/ - - 8/ +T.
aλ;
Now to enforce the simplex constraint, sum over all i:
(1s + 7) = =1 1 Ei 1 Wi
Thus, the projection without considering the non-negativity constraint is: X +,+ -يكيت we modify the solution by clipping negative values to zero and redistributing the remaining weight. This is efficiently handled by sorting v in descending order and determining a threshold т such that the projected vector remains non-negative.
If any X < O,
Algorithm 2: Euclidean projection onto A
Function ae (A R$) Sort À into
2 m2 2 ... n7
7 = max S j S t: th + } (1 Ei 17)) > 00}
y ツ= - (1- - Em1 mh:)
return Å = max A y, 0), i E (1,2,...,
4 Experiments & Results
Here, we present comprehensive empirical evaluation of the proposed Steady-State Bias Correction (SSBC- TD) algorithm across diverse benchmark domains. The experiments are designed to validate the method's effectiveness in mitigating steady-state distribution mismatch in off-policy TD learning with linear fune- tion approximation. We assess performance using Root Mean Square Error (RMSE) of value predictions: RMSE- = I|V Φx||2 against true value functions. All results are averaged over 10 independent runs to ensure statistical robustness. Key aspects evaluated include:
Generalization across domains: Discrete (Circle Chain (Prediction + Control), Gridworld Cliff Walking, Taxi) and continuous (Mountain Car, CartPole, Acrobot) state spaces
Hyperparameter scnsitivity: Impact of step-sizes (a4, Bi) convergence
Trajectory robustness: Performance under varying episodic path structures
31

---

Published in Transactions on Machine Learning Research (01/2026)
Distributional fidelity: Accuracy of Gaussian mixture approximations for stationary distributions
Diicoount factor and design distribution sensitivity: Impact of discount factor and design distribution on prediction error.
All environments are modified ensure ergodicity (e.g., respawning agents in terminal states) for well-defined steady-state distributions.
4.1
Discrete Domain
4.1.1
Circle Chain
We consider ring-structured Markov chain with 100 discrete states S = {0, ,...,, n......11111 and circular distance d(i.j) = min(li jl, n -jl.. Each chain is defined by transition kernel that moves Left or Right between neighbouring states while preserving irroducibility and apcriodicity. To ensure that the stationary distribution of each chain non-uniform yet unique, we construct the transition probabilities Gaussian distribution v on the ring.
satisfy detailed balance with respect desired wrapped-Gaussian
P3 = v(i +((+1) Pi+1.
iES.
This is implemented by choosing
Pa = min ktt)) }. P, = min o(i-1)) =1, PA,t1 Ps v(t)
yielding an irreducible, reversible, and uniquely stationary Markov chain with v as its stationary law. This Mctropolis -astpinls-Hastinns construction ensures that the induced stcady-state remains non-trivial even after full mixing. The target stationary distribution is fixed as wrapped Gaussian centred at 0:
(i) exp
d(i.0)²
20
while the behaviour chain's stationary distribution is its spatially shifted counterpart
(c) (i) exp d(t.c(p)))
c(p) = (1- n/2, P [0,1].
As p decreases from 1 to O, the mode of 2/ moves antipodally across the ring, generating controllable distributional mismatch between behaviour and target steady-states. The auxiliary design distribution q is uniform, approxinating the steady-state of a fully mixing kernel and representing an uninformative bias baseline. This choice reinstates the eeuah- treatment of states. The behaviour steady-state distribution v6")) is approximated by Gaussian mixture surrogate h with l = 10 wrapped components.
-
Figure 3a shows that SSBC-TD consistently attains lower RMSE than baseline off-policy TD yet remains above the on-policy limit, confirming effective bias mitigation without instability. Figure 3b highlights the structural discrepancy between the uniform design 9 and the localized target steady-state Vx, principal source of off-policy bias. Finally, Figure 3e demonstrates how the behavioural distribution v(") moves from orthogonal aligned as p increases, with the fitted mixture surrogate accurately tracking this progression. Together, these results verify that SSBC-TD maintains asymptotic consistency and achieves favourable bias variance trade-off as the steady-state overlap between behaviour and larget policies improves.
4.2 Control: Policy Optimization under Steady-State Bias Correction
We extend the steady-state bias correction framework from off-policy prediction control on the Circle- Chain MDP. As before, the environment compriscs
100 cyclically connected states and two actions A = (Left, Right). Rewards follow Gaussian centered at 0:
r(s) (2a
a, 5-=5, 5,
(87)
32

---

Published in Transactions on Machine Learning Research (01/2020)
3620
Off-policy (baseline)
0.05
Uniform
3600
S5BC-TD (ours)
Target steady state
policy
3580
0.04
Probability
RMSE
3540
0.03
3520
0.02
3500
0.01
3460
0.00
100000
200000
300000
400000
500000
0
40
80
100
Iteration
(a) RMSE evolution horizons; SSBC-TD interpolates between and off-policy TD.
(b) Uniform design distribution g(s) non-uniform get steady-state Py.
Target Ma
c.000
Mixture surrogate
p
0.00
p 0.33
0.04
Probability
0.02
0.00
p 0.66
p 11000
Probability
0.04
0.02
0.00
20
40
60
80
100
0
20
40
60
80
100
State Index
State index
(c) Evolution of xf0) and its mixture surrogate across e (0, 0.33, 0.66, 1}: the surrogate accurately tracks the shifting behavioural distribution.
Figure 3: Circle Chain experiments. (a) SSBC-TD achieves intermediate RMSE between on- and off-policy TD, miligaling steady-state bias while maintaining stability. (b) The uniform weighting q differs from vx, motivating reweighted updates. (c) As p increases, v/() progressively aligns with Pa, and the mixture surrogate (& 10) closely follows this transition, illustrating smooth bias reduction as overlap grows.
with discount factor = 0.9. The behaviour policy induces a stationary distribution vs concentrated near the antipodal region the reward peak (sR550), creating a strong distributional mismatch with the optimal policy's stationary measure vz The auxiliary design distribution uniform, q(s) 1/n, representing an uninformative baseline.
Rationale for uniform q: In classical dynamic programming (value policy iteration with known model), Bellman backups pointwise, and the contraction is measured in the || lloo norm, giving equal weight to all states. In contrast, trajectory-basee off-policy TD with function approximation minimizes a projected Bellman error under the behaviour steady-state geometry: Φ TE, (Φr - T~ (fx)) = O, which overweights frequently visited states and underweights others, inducing a steady-state bias. Replacing Ey by E, through multiplicative weights {(s) x (s))/((s) yiclds the fixed point of T,, Choosing uniform q restores the state-agnostic weighting of classical value or policy iteration, where all states contribute equally to the Bellman residual. This aligns off-policy TD with the unbiased model-based ideal, removing systematic
33

---

Published in Transactions on Machine Learning Research (01/2026)
skew from 26 while maintaining mumerical stability. Practically, uniform g serves as robust surrogate for the (unknown) larget steady-state distribution Vn; bridging trajectory-based and classical value-iteration perspectives.
Algorithmic procedure: Policy iteration alternates between value estimation (via naive off-policy TD or SSBC-TD) and greedy improvement using
Q=(s,a)= r(s,a) + E
(88)
where Te denotes the value-parameter estimate from the precoding evaluation phase and ((s) are the RBF features. The next policy updated
Th+i(a () 1aa arg max (s.
(89)
with deterministic tie-breaking. Each evaluation phase performs 2000 TD updates with the same feature representation (20 RBFs) and step-size schedule; results are averaged across 50 independent random seeds.
Figure Aa shows the discounted return across policy iterations. SSBC-TD attains faster and higher returns than naive off-policy TD, with narrower variability bands. This confirms that re-projecting the Bellman operator in (g) suppresses the instability caused by the skewed visitation frequencies v.. Figure 4b reports the RMSE between the estimated and true values v². The bias-corrected variant converges more rapidly and achieves lower steady-state error, verifying that geometric alignment with q mitigates the accumulation of long-horizon bias. Figure Ac visualizes the learned action probabilities P(Right s). The SSBC-TD policy exhibits a sharp, symmetric decision boundary that efficiently drives trajectories toward the reward centre, whereas the naive policy remains blurred and asymmetric, reflecting persistent influence from Pa.
Across all metrics, SSBC-TD obtains superior control behaviour- higher asymptotic returns, reduced RMSE, and a sharper optimal policy-while maintaining stability comparable to on-policy TD. By explicitly reweighting the evaluation step toward uniform g, the algorithm neutralizes steady-state bias from the behaviour chain and ensures that the policy-improvene the intended state weighting. Empirically, these findings confirm that steady-state correction not only re- stores consistency in value prediction but also enables bias-free off-policy control with markedly improved sample efficiency.
operator follows the true Bellman gradient under
4.2.1 Gridworld Cliff Walking
In the Gridworld Cliff Walking domain, the agent operates on a discrete 10 10 grid (IS) = 100) where the objective is to reach the goal while avoiding the cliff region. We set y 0.9 and employ behaviour policy biased toward the left side of the grid, producing left-skewed steady-state distribution vs. The target policy a prefers the safer right-side cells, inducing right-skewed stcady-state V,- The correction distribution q is uniform across states, providing a flat reference for bias correction. The surrogale h approximates 4b using a mixture of = Gaussian components over discrete state indices.
Figure 5a demonstrates that SSBC-TD reduces the steady-state bias substantially compared with the base- line off polico TD method, approaching the accuracy of on-policy learning while maintaining stability across iterations. The results confirm that steady-state correction yiclds lower asymptotic error and improved smoothness of convergence. Figures 55 5e illustrate the underlying distributional mismalch and its correc- tion. The uniform q(s) does not capture the asymmetry present V/, which leads to biased value estimation in standard off-policy TD. By contrast, the surrogate lig 3. sufficiently approximates v,, ensuring acceptable reweighting of the temporal-different uncertainty band observed for the mixture indicates that the surrogate fit is consistent across multiple runs. Overall, SSBC-TD achieves balanced trade-off between bias reduction and variance control, validating its robustness in environments with asymmetric occupancy distributions.
updates and stabilizing the projected Bellman operator. The narrow
34

---

Published in Transactions on Machine Learning Research (01/2026)
SSSC-TD eval (uriform otopo Iy
SSBC-TD eval tunifon qi
off-cclccy
Estimated discounted return
0.70
0.35
0.30
0.65
0.25
0.20
0.15
0.10
0.55
0.05
8
10
12
10
12
Policy iteration
Policy iteration
(a) Discounted return
(b) RMSE vs 0
SSRC-TD finel poicy
Neive TD finel policy
0.6
PeRight]
0.2
0.0
100
Stote Index
(c) Learned policy probabilities P(Right | s)
Figure 4: Circle Chain control results. (a) Discounted return policy iteration. (b) RMSE between estimated
and true values. (c) Final policy distribution P(Right s). SSBC-TD achieves higher asymptotic return, reduced steady-state error, and sharper optimal boundary compared to naive off-policy TD, demonstrating effective bias correction and improved sample efficiency.
4.2.2
Taxi Domain
The Taxi environment consists of a finite tabular state space S with [S] = 500 states encoding the taxi loca- tion, passenger position, and destination, and a discrete action set representing navigation and pickup/drop- off actions. We consider an off-policy prediction task where the behaviour policy To selects each action uniformly at random every state. Under this policy, the induced stationary distribution vs empirically close to uniform, and hence the steady-state bias that arises from evaluating the value function under vs instead of the evaluation distribution is inherently small.
We evaluate our algorithm with discount factor of 7 0.2. The behaviour stcady-state distribution is approximated by Gaussian mixture, where each component is truncated Gaussian distribution over the closed interval [o, 600). We set l. = 10 mixture components. The design distribution q is chosen to be uniform over all states, q(s) = 1/ ensuring that all states receive equal weight in the RMSE metric. The correction factor Çe then reweights TD updates so that their effective sampling distribution aligns with this uniform larget.
From Figure f both the base off- off-policy algorithm and our approach achieve nearly identical results. Because the uniform behaviour policy induces almost uniform stationary distribution across the 500 discrete stales, the mismatch between Pb and the evaluation distribution is minimal in the Taxi domain. Consequently, the uncorrected off-policy TD bascline already exhibits low bias, and all methods converge to similar asymptotic RMSE values. heless, Figure shows that SSBC-TD consistently tracks the on-policy TD curve while maintaining slightly higher variance due to its correction factor Ç, The improvement over the plain off-policy baseline indicates that even small distributional mismatches are effectively mitigated by the correction. The on-policy and SSBC-TD curves nearly coincide, confirming that the state-distribution correction produces on-policy-lin like behaviour in this domain. Figure 6b further illust that the learned surrogate hgg X
35

---

Published in Transactions on Machine Learning Research (01/2026)
0.030
Uniform
Target steady state vn
220
off-policy (base)
0.025
SSBCTD (our algorithm)
210
cn-palicy
Probabilitty
0.020
0.015
RM5E
200
0.010
0.005
0.000
100
State
0
100000
200000
300000
400000
500000
Baration
(a) Bias estimation (RMSE) va. iteration across algorithms. Shaded regions indicate one standard deviation.
(b) Uniform g()) va. target steady-state distribution Ve The target policy induces a right-skewed stationary occu- pancy measure, while g remains flat.
steady vs Mixture surrogate mean =5
0.020
Probability
0.015
0.010
100
State
(c) Behaviour steady-state rogate M(:;0",X" (f = 5). The mixture captures the left- skewed nature of P3 with uncertainty.
Caussian-mixture
Figure Gridworld Cliff Walking. (a) SSBC-TD consistently bridges the gap between on- and off-policy learning, achieving low bias and smooth convergence. (b c) The mismatch between q(s), Vr, and vs illustrates how steady-state bias arises and how the surrogate correction mitigates it.
closely approximates the true steady-state distribution Pb. The red mixture mean envelope remains nearly flat across the 500 discrete states, validating that the mixture model with = 10 components can faithfully represent an almost uniform distribution. Together, these results show that SSBC-TD stable and nearly unbiased when the steady-state distribution is uniform, matching on-policy accuracy while requiring only off off-policy samples.
4.3
Continuous Domain
4.3.1
Mountain Car
The Mountain Car environment consists of continuous two-dimensional state space (x, representing the car's horizontal position and velocity, and discrete action set {-1,0, +1} corresponding to left, no, right acceleration. The goal is to reach the top of the right hill despite insufficient engine power for a direct ascent, requiring the agent to build momentum by oscillating between the slopes. We evaluate our algorithm with discount factor = 0.4, emphasizing short-horizon control stability. The behaviour policy T selects actions uniformly, while the target policy prefers rightward thrust near the left slope and conservative deceleration near the goal region. The behaviour steady-state distribution in approximated online using Gaussian-mixture density with & = 15 components, and the design distribution q is taken to be flat over
36

---

Published in Transactions on Machine Learning Research (01/2026)
off-palicy (base)
102.5
SDBCTDD (our algorithm)
on-policy
100.0
0.0025
97.5
0.0020
95.0
Probabiility
0.0015
92.5
0.0010
87.5
0.0005
100000
200000
300000
400000
500000
0.0000
lteration
0
100
200
300
400
500
State
(a) RMSE Itcration (> 0.9)
(b) Steady-state Mixture surrogate (IS) 500,
10)
Figure 6: (Left) RMSE as a function of trajectory transitions, comparing on-policy TD, the off-policy TD baseline, and SSBC-TD. The on-policy and SSBC-TD
nearly coincide, both lying below the uncorrected off-policy baseline, indicating that the state-distribution correction achieves on-policy like accuracy under nearly uniform steady-state. (Right) Approximation the uniform steady-state distribution (black stems) by Gaussian-mixture surrogate hg. S- with = 10 over 500 discrete states, confirming that the mixture surrogate accurately matches the almost uniform v4.
the reachable region So = x [-0.07 0.07]. The correction factor Ço reweights updates toward this uniform distribution, mitigating stcady-state bias without amplifying variance excessively small 7.
Figure 7 shows the RMSE versus iteration. Each curve denotes the mean over independent runs. The lower discount factor accelerates the contraction of the projected Bellman operator, ensuring stable and monotonic error decay throughout training. The SSBC-TD trajectory lies consistently between the on-policy and off- policy curves, confirming that the state-distribution correction removes most of the steady-state bias while maintaining stable learning dynamics. At a low discount. factor (7 = 0.4), the value function depends primar- ily on near-term rewards, making the learning process less sensitive to long-horizon horizon distributional mismatch. Consoquently, all three methods converge smoothly, though off-policy TD remains slightly biased duc evaluation under the behaviour distribution vs. The SSBC-TD correction reduces this bias by reweighting updates toward the flat 9, thereby aligning the projected Bellman fixed point more equally likely across all the observed states. Since is small, the effective variance amplification caused by is modest, yielding a smooth error curve and stable asymptotic behaviour. Overall, the experiment demonstrates that SSBC-TD remains consistent and robust even when the target horizon is short, achieving a good balance between bias reduction and variance control.
4.3.2 CartPole
The Cartpole environment has a continuous four-dimensional state s = (xx i. 0,00 and a discrete two-action set {11 corresponding to left or right force applied to the cart. We compare the algorithm under the discount factor = 0.9. The behaviour policy मe selects each action uniformly at random, inducing a broad steady-state distribution 14 that explores a wide range of cart positions and pole angles. The target policy stabilizing controller that maintains the pole distribution Ma is concentrated around small |0| and [0]. We approximate using Caussian-mixtun density with f = 15 mixture components, fitted online from the behaviour trajectory via incremental updates. The design distribution is taken to be flat over bounded region So C S representing the empirically reachable portion of the state space, defined as the axis-aligned bounding box enclosing the 1st 1st-99th percentile of states encountered under Ty:
the upright configuration, whose stationary
So [Smin 8, 8max + 6).
((s) x 1(s E 5o).
with d providing small margin (5-10%) beyond observed extremes. For reproducibility under the standard Gym Cartpole, the analytical bounds 1-2.4,2.4 2.4) 3.00 [-0.21, 0.21) x (-3,5.3.5] were used as So.
37

---

Published in Transactions on Machine Learning Research (01/2020)
1.2
On-policy
0.8
SSBC-TD (ours)
Off-policy
0.6
RMSE
0.4
0.2
0.0
50
100
150
200
300
Iteration
Figure 7: Performance on the Mountain Car environment with discount factor = 0.4. The plot compares on-policy TD, SSBC-TD (ours), and the off-policy (baseline) terms RMSE versus iteration. Curves represent mean values across multiple runs, with shaded bands denoting empirical variability: narrow for on-policy TD and wider for SSBC-TD and off-policy TD. All methods converge smoothly, and the SSBC-TD curve lies consistently between on-policy and off-policy baselines, indicating effective steady-state bias correction.
This flat choice assigns equal weight to all reachable states, ensuring the RMSE metric is unbiased toward any region of high occupancy in 14. The correction factor Se reweights TD updates away from the behaviour steady-state and toward this uniform design distribution, while the importance ratio adjusts for action- selection mismatch. Figure g shows the RMSE versus iteration, where each curve denotes the mean over independent runs and shaded regions mark empirical variability. The performance hierarchy follows naturally from the bias-variance interplay among the three estimators. On-policy TD remains unbiased since sampling and evaluation distributions coincide. SSBC-TD introduces the correction factor Gr which reduces steady- state bias, but increases variance due to state-dependent reweighting. Consoquently, converges more slowly than on-policy TD yet achieves lower error floor than the uncompensated off-policy TD baseline, whose updates project onto the behaviour distribution and thus converge to biased fixed point. The advantage of SSBC-TD is more pronounced due to larger 7, since long-horizon dependencics amplify steady-state bias effects.
1.2
... On policy TD (ours) Off-policy TD (baseline)
1.0
0.8
RMSE
0.6
0.4
0.2
0.0
100
150
200
250
300
Iteration
Figure 8: Performance on the Cartpole environment with discount factor 7 = 0.9. The plot compares on-policy TD, the proposed SSBC-TD, and the off-policy TD baseline in terms of RMSE versus iteration. Curves show mean values across independent runs; shaded bands represent empirical variability. All three methods display non-monotone transient dynamics, but ultimately settle into stable ordering in which on-policy TD attains the lowest asymptotic error, SSBC-TD converges to an intermediate level, and the off-policy TD baseline retains the highest residual RMSE, consistent with partial correction of steady-state bias using the user-specified Gaussian g()).
38

---

Published in Transactions on Machine Learning Research (01/2026)
4.3.3 Acrobot
The Acrobot environment consists of two-link underactuated pendulum with continuous state (01, 02. 02,01, f2), 02), where 0 and 02 denote the angles of the first and sccond links relative to the vertical downward direction, and 0,, 02 are their respective angular velocities. The action space is discrete, A = {-1, 0, +1]; corresponding to the application of negative, zero, positive torque at the joint between the two links, We consider a behaviour policy %. that selects cach action uniformly at random, To(a s) = 1/3, thereby inducing a broad steady-state distribution is that covers both swing-down and transitional configurations. The target policy æ is a hand-crafted stochastic stabilizing controller defined by a linear feedback rule: a continuous torque signal
181 81
i1 02
(*(s) - Kg
K3
82 85
where (0,,8) 85) corresponds the upright equilibrium. The resulting torque is discretized to the nearest action in {-1, 0, +1}, executed with probability 1-e, E, and replaced by random alternative action with probability This rule ensures persistent exploration while maintaining stabilizing behaviour around the upright configuration. The induced Markov chain under admits a unique stationary distribution Vx that is sharply concentrated near the upright manifold. Here, is chosen as a single Gaussian centered at the upright, configuration my = (0), 02, 0,)) with small angular variance and moderate velocity variance. This q is not required to equal Mx, but is intended to be qualitatively close, thereby biasing the iterates toward states that are important under the target policy. Also, we consider the mixture width to be f 100. Figure g reports the RMSE of value prediction versus iteration for on-policy TD, the off-policy TD baseline, and the proposed SSBC-TD, averaged over multiple independent runs. During the transient phase, all methods exhibit non-monotone fluctuations as both the density model h and the value parameters adapt under noisy off off-policy data. Eventually, a consistent ordering emerges: on-policy TD achieves the lowest asymptotic RMSF, as it directly samples from the true stationary distribution ux; SSBC-TD converges to an intermediate error floor that lies strictly below the uncompensated off-policy baseline, demonstrating that the Gaussian q is sufficiently close to Vx to mitigate steady-stale bias; and the off-policy TD baseline retains the highest residual error due to its dependence on vs without long- nonghorizon correction. This behaviour aligns with thcoretical expectations: when q approximates but does not exactly match the target stcady-state distribution, the proposed correction reduces but cannot eliminate the asymptotic bias, thereby narrowing, but not closing the gap to on-policy performance. We also provide here the likeliness between the behaviour policy steady-state probability distribution vs and the estimated surrogate distribution h(-; 0- X²).. The results are provided in Figure 10
4.4
Sensitivity
We empirically examine how the two timescale parameters, the TD step-size as and the importance-ratie step-size 3r, influence the accuracy and stability of our SSBC TD algorithm. All experiments are conducted on the classic control task MountainCar Car from the Gymnasium suite. Our hyperparameter grid search reveals a critical interplay between the TD step-size (a,) and the ratio step-size (3.). The heat map (Figure I1 illustrates that the best-performing configurations cluster around higher a values, particularly when paired with smaller &e values. The combination of a₁ = 0.2 and 3. = 0.005 achieves the lowest final RMSE, indicating an effective balance between rapid updates the steady-state distribution approximation and gradual correction of the steady-state distribution mismatch. This result underscores the importance of carefully tuning these timescales to mitigate the "deadly triad" interaction between function approximation, bootstrapping, and off-policy learning. The curve plot (Figure 12) further validates this obscrvation by showing how different. Or seltings converge over episodes. Notably, the curve for a, = 0.2 exhibits the fastest decline in R.SE, stabilizing at the lowest error level compared to other configurations. Polyak averaging plays a crucial role in smoothing out fluctuations during training, as evidenced by the reduced variance the RMSE curves across episodes. By incorporating Polyak averaging, our approach effectively mitigates the noise intuude by stochastic updates, leading to more stable and accurate value predictions.
39

---

Published in Transactions on Machine Learning Research (01/2026)
Off-policy (baseline)
2.00
SSBC-TD (ours)
On-policy TD
02
1.50
RMSE
1.25
1.00
01
0.75
0.50
0.25
0.00
0
25
50
75
100
125
150
175
Iteration
Figure 9: Performance on the Acrobot environment for discount factor = 0.8. The plot compares on-policy TD, the proposed SSBC-TD, and the off-policy TD baseline in terms of the root-mean-squared error (RMSE) of value prediction versus iteration. Each curve represents the mean across multiple independent runs. All methods exhibit transient non-monotone phases early in training as the density model h(s; 6, A) and the value parameters co-evolve. As learning stabilizes, the on-policy TD achieves the lowest asymplotic error, SSBC-TD converges to intermediate error floor, and uncompensated off-policy TD baseline attains the highest residual RMSE. This hierarchy confirms that the state-distribution correction using the user-specified Ganssian g(x) effectively mitigates, but does not completely remove the steady-state bias inherent in off-policy prediction.
Origital Steady State Distribution probabii:iis
Indivicual and comabined Caussians Optimised g
larget
6.025
0.012
0,020
0.010
c2mblned
0.000
0.010
0.005
0.002
0.000
0
40
80
80 100
100
sdate
Figure 10: Comparison between the true behaviour-poliey steady-state distribution and its surrogate mixture estimate. Left: Empirical steady-state probabilities of the behaviour policy (14) computed over multiple long- rum trajectories. Right: Individual Gaussian components and their optimally weighted combination forming the surrogate distribution hn. x. The close aligument between the mean surrogate and the empirical steady-slate profile demonstrates that the fitted mixture provides a close and stable approximation to the behavionr-policc stationary distribution.
4.5 Trajectory Robustness
We evaluate trajectory-length sensitivity on a 5 Gridworld with absorbing terminals at (0,0) and (4, 4). Each episode starts in the cell (0, 4) and evolves for 60 time-steps under a uniform random behaviour policy; upon reaching terminal, the agent is reset to the start cell and the trajectory continues. We generated five such trajectories (Figure 13 Run 0-4), using independent random seeds to expose path-level variability. We then applied a constant step-size SSBC-TD agent with Br = 0.05. The micro-trajectories illustrate how an ergodic uniform policy can still visit states in markedly different orders at short horizons. Run 1 drifts almost exclusively downwards, Run performs horizontal sweeps along the top row before descending, whereas Run forms an almost symmetric lattice tour. Runs 0 and 4 highlight the "reset effect": diagonal sprint to the lower left corner followed by immediate reinitialization injects additional exploratory diversity
uniform-random
40

---

Published in Transactions on Machine Learning Research (01/2026)
95
90
85
8.t
80
0.02
75
0.025
0.050
0.075
0,100
0.125
0.150
0.175
0.200
step-size)
Figure 11: Hyperparameter sensilivity of SSBC-TD MountainCar: Final RMSE function of the TD step-size (au) and ratio step-size (5t). Darker shades indicate lower error. The minimum RMSE (marked) occurs at = 0.2 and B. 0.005, revealing that aggressive TD) updates paired with conservative ratio estimation optimally balance convergence and stability.
Best B_t for each a_t (grid features, C=5)
100
90
RMSE
85
80
a_t=0.02, t_t=0.005
75
a_t=0.05, 8_t=0.005
a_t=0.1, B 2.t=00005
a_t=0.2, t.000
25
75
100
125
150
175
200
Episode
Figure 12: Convergence of SSBC-TD on MountainCar using the best 3 for each o, (from Figure 11
The = 0.2
curve (8,- 0.005) achieves the fastest error reduction and lowest asymptotic RMSE (ss 70), demonstrating Polyak averaging's 's role in stabilizing high-step-size size regimes, Smaller au (e.g., 0.02) exhibit slower convergence due to delayed implicit averaging.
41

---

Published in Transactions on Machine Learning Research (01/2026)
Distinct Grid Worddorrd Trajectories (One per Run)
0
1
0
2
0
Figure 13: Trajectories under uniform-random behaviour policy.
Run
Run
Run
Run
Run
0.4
044
Densto
0.3
Denety
0.1
Figure 14: Comparison of the approximation of the true behaviour policy's stationary distribution using the Gaus
sian mixtures across various trajectories
Uniform RMSE
2
0
1
2
3
4
Run
Figure 15: SSBC-TD demonstrates trajectory-agnostic stability, efficiently correcting steady-state bias regardless of path stochasticity
that would otherwise take longer to accumulate. These early visitation biases explain the modest run-to-run spread observed in the residual histograms: SSBC-TD first adapts to whichever subset of states it samples most frequently. Over the long run updates, however, the random policy's 's mixing property smooths out those disparities: each run ultimately visits every state with frequency close to the stationary distribution, and the TD iterates converge to a common, tight error band (uniform-RMSE x 4.1..
4.6 7 Sensitivity
Here we study the relationship between the discount factor 7 and the error in off-policy value prediction. The experiment considers a Taxi discrete control lask. The stale and action spaces are discrete. Both TD(0) and SSBC-TD are trained off-policy using the same fixed behaviour- target policy pair and identical feature reprosentations, with final RMSE computed against Monte Carlo estimates under the target policy. The results are illustrated Figure 16 As y increases, the longer effective horizon exacerbates distribution mismatch and steady-state bias in plain TD(0), leading to steady rise in RMSE and greater variability across
42

---

Published in Transactions on Machine Learning Research (01/2026)
pftiplicy TDXO)
SSBC/TD (c-omrected)
1.6
1.4
RMSF
1.2
1.0
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Discount factor
Figure 16: RMSE as function of discount factor for plain off-policy TD(0) and steady-state bias corrected TD (SSBC-TD) discrete Taxi control task. SSBC-TD consistently achieves lower RMSE, with the gap widening increases.
choice oratl unfarm 9g Ibehavlor'
2.0
RASE
1.4
1.2
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Discount factor
Ablation on the choice of stationary target for SSBC-TD a Taxi control task. Four variants considered: oracle V4 (true target policy distribution), uniform over states, behaviour visitation distribution vs, and a deliberately misspecified distribution. Closer alignment of q to us yields consistently lower RMSE, with the advantage widening as ? 1.
Figure 17:
runs. SSBC-TD, by applying (-weights that approximate the stationary distribution correction, mitigates this bias and maintains a lower error profile across values. The widening advantage of SSBC-TD for + 1 aligns with the thcoretical predictions: the stability condition k3>² for plain TD becomes harder satisfy high y, whereas SSBC-TD effectively reduces the mismatch constant, relaxing the condition 2g < 1, thereby extending the range of stable and accurate operation.
43

---

Published in Transactions on Machine Learning Research (01/2026)
4.7 q Sensitivity
We study here the sensitivity of SSBC-TD to the choice of g, the sialionary-ddittributton targel. used in its correction term. The discrete control task used here is a Taxi environment. We consider a discrete control task with fixed target and bohaviour policics, identical feature representation, and the same step-size schodules across all runs. The SSBC-TD algorithm is applied with four different choices the stationary-distributtien target q its correction term G-- The oracle choice uses the exact stationary distribution ux of the target policy; uniform assigns equal probability to all states; behaviour uses the empirical stationary distribution vs from the behaviour policy; and mis-specified uses a biased distribution that incorrectly overweights rarely visited states and underweights important ones. The RMSE 7 curves show a clear ordering: the oracle q achieves the lowest error across all 7 with the margin over other choices growing as y increases. This matches the theoretical prediction that the stability condition improves from kay² < 1 to KgHt,7² <1, where Kgtig is minimized when q = Vz- Uniform g performs moderately well for smaller y but suffers at large y due equal weighting of states that are rarely relevant to the target policy. Using 24 offers only limited improvement, since it does not correct the long-horizon mismatch. The mis-specified q provides the smallest benefit and, for high 7 behaves similarly to plain TD, illustrating that poor q choices can erase the gains of the correction. This underscores the importance of accurate or well-chosen for leveraging SSBC-TD's stability advantage in long-horizon scttings.
5 Conclusion & Future Work
In this paper, we consider the off-policy value prediction in reinforcement learning, specifically in the context of linear function approximation ion. The proposed algorithm aims to minimize the steady-state bias in the off pollcy value prediction, where the bias arises due to the differences in the sampling distribution of states and actions between the target policy and the behaviour policy. Our work opens up several avenues for future research. First, integrating steady-stale bias correction with deep value function approxinators is promising direction to tackle large-scale problems. Second, the idea of distribution correction might be extended to control settings: for example, off-policy actor-critic algorithms could use similar mechanism to reweight the critic updates, or one could correct state occupancy in off- policy gradient methods. Third, an interesting theoretical question is how steady-state bias correction interacts with function approximation error and whether can alleviate the deadly triad (function approximation, off off-policy, and bootstrapping).
References
Leemon Baird et al. Residual algorithms: Reinforcement learning with function approximation. In Proceed- ings of the twelfth international conference on machine learning, pp. 30-37, 1995.
Dimitri Bertsekas. Reinforcement learning and optimal control. Athena Scientific, 2019.
Vivek S Borkar. Stochastic approzimation: dynamical systems viewpoint, volume 9. Springer, 2008.
Lawrence D Brown. Fundamentals of statistical exponential families: with applications in statistical decision
theory. Ims, 1986.
Yash Chandak, Scott Niekum, Bruno da Silva, Erik Learned-Mille Emma Brunskill, and Philip S Thomas Universal off-policy evaluation. Advances in Neural Information Processing Systems, 34:27475-2744 27490, 2021.
Peter W Glynn and Donald L Iglehart. Importance sampling for stochastic simulations. Management science, 35(11), 1989.
Assaf Hallak and Shic Mannor. Consistent on-line off-policy evaluation. In International Conference on Machine Learning, pp. 1372 1383. PMLR, 2017.
Yongming Huang, Chunmei Xu, Cheng Zhang, Meng Hua, and Zhengming Zhang. An overview of intelligent wireless communications using deep reinforcement learning. Journal of Communications and Information Networks, 4(2):15-22) 29, 2019.
44

---

Published in Transactions on Machine Learning Research (01/2026)
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In Interna- lional conference on machine learning, pp. 652 661. PMLR, 2016.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions Intelligent Transportation Systems, (3(6):490944922 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances neural information processing systems, 33:1179-1191, 2020.
Jongmin Lee, Wonscok Jeon, Byungjun Lee, Joelle Pincau, and Keo-Eung Kim. Optidice: Offlinc policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pp. 6120 6130. PMLR, 2021.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. Aduances in Newral Information Processing Systems, 31, 2018.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. arXiv preprint arxiv: 1904109444..... 2019.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation
across representations with applications educational games. AAMAS, volume 1077, 2014.
Scan Meyn. Control systems and reinforcement learning. Cambridge University Press, 2022.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Bchavior-agnostic cstimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019.
Boris Teodorovich Polyak. A new method of stochastic approximation type. Avtomatika i telemekhanika, (7):98-107, 1990.
Doina Precup, Richard S Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. 2000.
Doina Precup, Richard Sutton, and Sanjoy Dasgupta. Off-policy temporal-difference learning with function approximation. In ICML, pp. 417-424, 2001.
Martin L Puterman. Markow decision processes: discrele stochastic dynamic programming. John Wiley &
Sons, 2014.
Ralph Tyrell Rockafellar. Convex analysis, volume 28. Princeton university press, 2015.
RY Rubinstein. Simulation and the monte carlo method. new york, ny, usa: John wiley&sons, 1981.
David Ruppert. Efficient estimations from slowly convergent robbins-monro process. Technical report,
Cornell University Operations Research and Industrial Engineering, 1988.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv.1506.02433, 2015.
David Silver, Thomas Hubert, Julian Schrittwieser, Antonoglou, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362( 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Hamid Reza Maci, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In Proceedings of the 26th annual international conference machine learning, pp. 993 1000, 2009.
Richard S Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off- policy temporal-difference learning. The Journal of Machine Learning Research, 17(1):2600 2631, 2016.
45

---

Published in Transactions on Machine Learning Research (01/2026)
Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly robust bias reduction in infinite horizon off-policy estimation. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
Miguel Tejedor, Ashenafi Zebene Woldaregay, and Fred Godtliebsen. Reinforcement learning application in diabetes blood glucose control: A systematic review. Artificial intelligence in medicine, 104:101836, 2020.
Surya Tokdar and Robert Kass. Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics, 2(1))54-60, 2010.
J.N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation.
IEEE Transaclions on Aulomalic Control, 42(5):674 690, 1997. doi: 10.1109/9.58087
Cameron Voloshin, Hoang M Le, and Yisong Yue. Empirical analysis of off-policy policy evaluation for reinforcement learning. In Real-world Sequential Decision Making Workshop al TCML, volume 2019, 2019.
Weiran Wang and Miguel Carreira- Perrinéa.. Projection onto the probability simplex: An efficient. algo- rithm with a simple proof, and an application. arXiv preprint Xrv: 1541, 2013.
Yue Wang, Wei Chen, Yuting Liu, Zhi-Ming Ma, and Tie-Yan Liu. Finite sample analysis of the gid policy evaluation algorithms in markov setting. Advances in Neural Information Processing Systems, 30, 2017.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Newral Information Processing Systems, 33:6551-6561, 2020.
Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, and Irit Hochberg. En- couraging physical activity in patients with diabetes: Intervention using reinforcement learning system. Journal of medical Internet research, 19(10): 2017.
Yu. On convergence of emphatic temporal-difference learning. In Peter Grünwald, Elad Hazan, and Satyen Kale (eds.), Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pp. 1724-1751, Paris, France, 03-06 Jul 2015. PMLR.
Huizhen Yu. Least squares temporal difference methods: An analysis under general conditions. SIAM
Journal on Control and Optimization, 50(6):3310 3343, 2012. doi: 10.1137/100007799
Huizhen Yu. On convergence of some gradient-basod temporal-difforences algorithms for off-policy learning. 09652, 2017.
Assaf J Zeevi and Ronny Meir. Density estimation through convex combinations of densities: approximation and estimation bounds. Neural Networks, 10(1):99-109, 1997.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized oflline estimation
of stationary values. In International Conference on Machine Learning, pp. 11194-11203. PMLR, 2020.
46


Reviews:
[{"review_id": "XOe3LYRBho", "reviewer": "Reviewer_FWmD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I apologize for my late review. Please find my comments below.\n\nThe paper studies off-policy value prediction in RL. The problem, stated plainly, is that one generally wishes to compute the value function of a certain desired target policy, but is only given access to trajectories by a behavior policy which may differ in the steady-state. Importance sampling only corrects the single-step dynamics, the steady state may differ in a more sustained manner which calls for a more systematic approach. Figure 1 in the manuscript conveys this idea well.\n\nThey estimate the steady-state distribution of the behavior policy with a Gaussian mixture model and natural exponentiated family, and uses an importance-sampling-like approach to mitigate the bias. Specifically, their algorithm, Steady-State Bias Correction (SSBC), does an importance sampling-like correction to the standard temporal difference (TD) update. The paper shows theoretical guarantees of this algorithm along with experiments to numerically verify the results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "All the claims are backed with rigorous theoretical results and proofs. Upon taking a close look at the proofs, I did not find any errors.", "audience_yn": "Yes", "audience_explanation": "The paper solves an interesting problem in the field. While it is essentially a variant of importance sampling, the analyses provided is strong and backs up the claims in the paper.", "requested_changes": "- I would encourage the authors to use \\citet, \\citep for inline citations,\n- In the second last paragraph of page 3, the authors say that the computational complexity of computing $V_\\pi$ is $O(n^3)$. Technically speaking, this should be $O(n^\\omega)$, where $\\omega \\leq 2.372$ is the exponent of matrix multiplication,\n- It would be great if the authors could specify how they discretized the continuous environments in their simulations (they allude to this, but not explicitly, in the manuscript)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z27TNbYm1n", "reviewer": "Reviewer_q7Tc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an algorithm for off-policy value prediction with linear function approximation, which mitigates the steady-state bias by correcting the steady-state distribution discrepancies. The method first estimates the steady-state distribution of the behavior policy with a Gaussian mixture model, then uses the ratio between the probability of a given target distribution and this estimated probability to correct steady-state bias in the TD update. The paper provides theoretical guarantees on the value estimation error and demonstrates empirical performance on several environments. \n\n**Strengths**\n1. The off-policy evaluation problem addressed in this paper is an important area in reinforcement learning.\n2. The paper provides theoretical proof for key intermediate conclusions such as distribution approximation error, and presents an error bound for the predicted value function. \n\n**Weaknesses**\n1. The proposed algorithm relies on a predefined target distribution $q$, yet in most off-policy evaluation settings, the target policy's true stationary distribution is unknown.\n2. The main experiments set $\\gamma=0.1$, which cannot demonstrate the policy evaluation ability in long-horizon scenarios. Additionally, in most experiments, the standard deviation across independent runs is not reported. \n3. The main error bound in Theorem 6 is for $\\lVert \\Phi x_c^{TD} - V^\\pi \\rVert_{\\nu_b}$ rather than $\\lVert \\Phi x_c^{TD} - V^\\pi \\rVert_{q}$. Adding a bound that relates $\\lVert \\Phi x_c^{TD} - V^\\pi \\rVert_{q}$ to the RHS via $\\max_s \\frac{q(s)}{\\nu_b(s)}$ would make the conclusion clearer.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, the proposed algorithm and theorems are well supported by theoretical proofs and empirical experiments.", "audience_yn": "Yes", "audience_explanation": "The paper tackles the off-policy value prediction problem in reinforcement learning, an important problem that fits TMLR’s scope.", "requested_changes": "1. (Critical) There are a few places where the paper is not rigorous. For example, in the discussion after Theorem 3, reducing the kernel bandwidth $\\sigma$ will loosen the approximation rather than tighten it, as indicated by Theorem 3 and Figure 3. In Equation 24, it should be $\\leq$ rather than $=$ according to Lemma 1.\n2. (Strengthen) The main experiments should also test a larger $\\gamma$ value. More baselines should be included, and standard deviations across independent runs should be reported. \n3. (Strengthen) More detailed discussion of how this algorithm can be used when the target policy's steady-state distribution is unknown should be provided. Additionally, an error bound on $\\lVert \\Phi x_c^{TD} - V^\\pi \\rVert_{q}$ should be included in Section 3.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GBlZfbh0PB", "reviewer": "Reviewer_X7Kr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the problem of off-policy value function estimation in reinforcement learning with linear function approximation. The key challenge mentioned by the paper is steady-state bias where the learned value function parameter converges to a solution projected with respect to the behavior policy's stationary state distribution rather than the target policy's. The paper proposes an algorithm named Steady-State Bias Correction (SSBC) which introduces a second importance sampling-style correction to the standard temporal difference (TD) update (Equation (33)). This correction factor is the ratio of a predefined user-chosen state distribution, q(s), to an online estimate of the behavior policy's stationary distribution. The distribution h(s) is modeled as a Gaussian Mixture Model (GMM) and its parameters are learned concurrently with the value function via stochastic gradient ascent on a faster timescale. Experiments report RMSE of value prediction across small discrete/continuous control tasks, with ablations on stepsizes, trajectories, discount factor, and the choice of q.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- Overall, the theory seems reasonable and confirms the intuition that if one has a good model of the distribution mismatch (h) and a good target to correct towards (q), one can fix the problem. The other challenge which still remain is the practical difficulty of obtaining these good models and targets.\n\n- Consistent On-Line Off-Policy Evaluation Hallak & Mannor, 2017 seems to have looked at similar problem (\"distribution-mismatch bias in off-policy TD with function approximation\") earlier but is not cited in the paper. Please mention the similarities or differences with their proposed approach.", "audience_yn": "Yes", "audience_explanation": "The paper considers an important problem.", "requested_changes": "- The writing of the paper can be improved significantly. The paragraphs and sentences are too long making it harder to read.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XA6v2JoNI1", "reviewer": "Reviewer_Hwf3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the distribution mismatch problem in off-policy value prediction using linear value function approximation. The first contribution of the paper is an off-policy TD error bound relating approximation error to distribution shift and discount factor (theorem 1 & 2). This result motivates the proposed algorithm, where a policy distribution correction ratio and a marginal state distribution correction ratio are added to the standard TD learning update rule. An error bound is provided for this algorithm. The numerical experiments are conducted on a set of small discrete environments and discretized continuous environments such as mountain car and cartpole. The proposed algorithm is shown to produce lower value error than both regular off-policy and on-policy. Some ablation experiments are conducted to validate the theoretical analysis.", "claims_and_evidence_yn": "No", "claims_explanation": "As I detail in the \"requested changes\" section, the paper is lacking some experimental details, making the results hard to judge. I'll be happy to revise this answer if my questions are clarified.", "audience_yn": "Yes", "audience_explanation": "I believe the off-policy error bounds, especially theorem 1, is of value to the RL community. However, I am not confident whether or not similar results have been published before.", "requested_changes": "Questions/comments:\n* In (4), could you specify what $\\Xi$ represents?\n* The authors discussed choosing the target distribution $q(s)$ as a reasonable proxy for the target policy. But this could be a pretty strong assumption given the behavior of the target policy is most realistically unknown. Otherwise there's no need to evaluate it.\n* Is there a way to compare the error bound in theorem 1 and theorem 6?\n* In the off-policy evaluation setting, you already have data from the behavior policy. Why using a two-time scale learning method instead of a two-stage learning method, where you first learn the marginal distribution of the behavior policy, freeze that distribution, and then do TD learning separately? In that case you don't need to tune the relative scale of the two learning rates. \n* I believe the authors should add more details about the experiments. Please see the next 3 questions.\n* The authors mentioned discretizing the continuous environments. Did you still use features to represent the value functions or the value functions are tabular? If the latter then the theory and experiment are not really matching?\n* For a number of environments, the authors randomly chose the target and behavior policy. I believe this potentially makes the problem trivial. For example, in mountain car, a random policy is unlikely to receive any reward, making value prediction the problem of predicting a constant. \n* In most plots, the authors showed value error by horizon/trajectory length. Is horizon/trajectory length the number of environment steps that the behavior policy was rolled out to collect data? If the behavior policy is stochastic, the data could be quite random/vary a lot between each rollout? If that's the case, why the green and red learning curves are so smooth? Shouldn't you repeat the experiment a few times for each configuration?\n* (Minor) At the beginning of section 4, you said \"The discount factor is fixed at γ = 0.1 universally\". This seems way to small for standard applications, where the discount factor is usually set between 0.9 and 0.99?\n* (Minor) In section 4.6, the authors could add a reference to figure 18.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)