Natural Policy Gradient for Average Reward Non-Stationary Reinforcement Learning
Authors: Neharika Jali
Eshika Pathak
Pranay Sharma
Guannan Qu
Gauri Joshi
Venue: Accepted by TMLR ()

Abstract: We consider the problem of non-stationary reinforcement learning (RL) in the infinite-horizon average-reward setting. We model it by a Markov Decision Process with time-varying rewards and transition probabilities, with a variation budget of $\Delta_T$. Existing non-stationary RL algorithms focus on model-based and model-free value-based methods. Policy-based methods despite their flexibility in practice are not theoretically well understood in non-stationary RL. We propose and analyze the first model-free policy-based algorithm, Non-Stationary Natural Actor-Critic, NS-NAC, a policy gradient method with a restart based exploration for change and a novel interpretation of learning rates as adapting factors. Further, we present a bandit-over-RL based parameter-free algorithm, BORL-NS-NAC, that does not require prior knowledge of the variation budget $\Delta_T$. We present a dynamic regret of $\mathcal{\tilde{O}} (|\mathcal{S}|^{1/2}|\mathcal{A}|^{1/2}\Delta_T^{1/6}T^{5/6})$ for both algorithms under standard assumptions, where $T$ is the time horizon, and $|\mathcal{S}|$, $|\mathcal{A}|$ are the sizes of the state and action spaces. The regret analysis leverages a novel adaptation of the Lyapunov function analysis of NAC to dynamic environments and characterizes the effects of simultaneous updates in policy, value function estimate and changes in the environment.

Published in Transactions on Machine Learning Research (02/2026)
Natural Policy Gradient for Average Reward Non-Stationary Reinforcement Learning
Neharika Jali, Eshika Pathak', Pranay Sharmal, Guannan Qu, Gauri Joshi
Carnegie Mellon University
Enjali, gqu, gourij) Sandrew.cmu.cdu, cmu.edu, 2pathak2@illimiii.. edu,, pranaysh@titb.ac.kk ac.in
Reviewed on OpenReview: https: opt/r/veenrr.....mmm net/ forum? id- ABJYNAYto0
Abstract
We consider the problem non-stationary reinforcement learning (RL) in the infinite-horizon average-reward setting. We model it by a Markov Decision Process with time-varying rewards and transition probabilities, with variation budget. of År- Existing non-stationary RL algorithms focus model-based and model-free value-based mothods. Policy-based methods despite their flexibility in practice are not theoretically well understood in non-stationary RL. We propose and analyze the first model-free policy-based based algorithm, Non-Stationary Natural Actor-Critic (NS-NAC), a policy gradient method with knowledge of the variation budget, a restart based exploration for change and novel interpretation of learning rates as adapting factors. Next, eliminating the requirement. of apriori knowledge of variation budget ÅT, we present a bandit-over-Ie RL based parameter-free algorithm BORL-NS-NAC, We present dynamic regret ((()) ///55/0 for both algorithms under standard assumptions, where T the time horizon, and [S), |AA are the sizes of the state and action spaces. The regret analysis leverages a novel adaptation of the Lyapunov function analysis of NAC to dynamie environments and characterizes the effects of simultaneous updates in policy, value function estimate and changes in the environment.
1 Introduction
While Reinforcement Learning has traditionally been studied in stationary environments with time-invariant rewards and state-transitions, this may not always be the case. Consider the examples of a carbon-aware datacenter job scheduler that tracks the dynamic electricity prices and local weather patterns [ and recommendation systems with evolving user preferences [2]. Time-varying environments are also observed in healthcare 3 ride-sharing 4. and multi-agent systems [5. Motivated by these applications, we consider the problem of non-stationary reinforcement learning, modeled by Markov Decision Process with time-varying rewards and transition probabilities, in the infinite horizon average reward setting. While many works consider discounted rewards 3
6 77,, the more challenging average-reward setting is vital in problems where the importance of rewards does not decay with time g I0 [I]. The key challenges for an agent operating in a dynamic environment are learning an optimal behavior policy that varios with the environment, devising an efficient exploration strategy, and effectively incorporating the acquired information into its behavior.
Current algorithms designed for non-stationary MDPs in the average reward setting can be classified broadly into modcl-based and model-free value-based methods. Model-basod solutions incorporate sliding windows, Model-
forgetting factors, and confidence interval managereent mechanisms into UCRL 12
13
14]
15
free value-based methods assimilate restarts and optimism into Q-Learning [6 17 and L.VII
18 19.
A
significant gap in literature is absence of model-free policy-based techniques for time-varying environments.
CCurently University of Illinois Urbana- Champaignn
(Currently C-MInDS, Indian Insitute of Technology Bombay

---

Published in Transactions on Machine Learning Research (02/2026)
The inherent flexibility of policy-based algorithms makes them suitable for contimuous state-action spaces, efficient parameterization in high-dimensional state-action spaces, and enables effective exploration through stochastic policy learning 20
Our Contributions. We tackle the problem of non-stationary reinforcement learning in the challenging infinite-horizon average reward setting in the following manner.
1. We propose and analyze Non-Stationary Natural Actor-Critic (NS-NAC), policy gradient method with knowledge of the variation budget, a restart based exploration for change and a novel interpretation of lcarning rates as adapting factors. To the best of our knowledge, this is the first modcl-free policy-basod method for time-varying environments.
2. We present a bandit-over-RI, based parameter-free algorithm BORI-NS-NAC that climinates the need for knowing the variation budget Ar apriori.
3. We present Ô (187//"/41)>> /5/6) dynamic regret bound for both algorithms under standard sumptions including uniform ergodicity where T is the time horizon, Ar represents the variation budget of rewards and transition probabilities, |S||A] is the size of the state-action space and Ō(-) hides logarithmic factors. The regret analysis leverages a novel adaptation of the Lyapunov function analysis of NAC dynamic environments and characterizes the effects of simultaneous updates in policy, value function estimate and changes in the environment.
2 Related Work
Non-Stationary RL. Solutions to the non-stationary RL problem can be categorized into passive and active methods. Active algorithms are designed to actively detect changes in the environment in contrast to passive ones which implicitly adapt to new environments without distinct recognition of the change. While we focus our attention on passive techniques with dynamic regret as the performance metric in this work, a comprehensive survey can be found in [1 and 222. Model-based solutions the infinite horizon average reward setting incorporate into UCRL a sliding window or a forgetting factor for piecewise stationary MDPs restarts [ and bandit based tuning of sliding window and confidence intervals 12
[4, variation for gradual or abrupt changes constrained by variation budget.
In the episodic setting, model-free value based methods assimilate restarts and optimism into Q-Learning [6] LSVI [8 19 and sliding window and optimistic confidence set based exploration into a value function approximated learning [7]. Further, in the episodic sctting, 23 proposes strategically pausing learning as an elfective solution to non-stationariity with forecasts of the future. 2 proposed an algorithm agnostic black-box approach that finds a non-stationary equivalent optimal regret stationary MDP algorithms. 12 T6 also present parameter-fres non-stationary RL algorithms that leverage the bandit-over-Ri RL framework
to adaptively tune algorithm without knowledge of the variation budget. Further, [6] presents information theoretic lower bound on the dynamic regret and 25 captures the complexity of updating value functions with any change, We note the distinction between the scope of this work and the body of research on adversarial MDPs which often allow for only changes in rewards, study the static regret and work with full information feedback instead of bandit. feedback. See Section A for a table of comparison of regret bounds.
Non-Stationary Bandits. A precursor non-stationary RL, the multi-armed bandit problem with time- vare-varying rewards was first proposed in 28 Solutions include UCB with sliding window discounting factor [6]] UCB with adaptive blocks of exploration and exploitation 27], Expp3 p7 Thompson Sampling with discounting factor [8] and bandit based sliding window tuning 29. Further, while most existing works assume arbitrarily (constrained by variation budget) changing reward distributions, 30 achieves an improved regret when the reward distributions change smoothly. Recent work by [T points out ambiguitics in the definition of non-stationary bandits and how the dynamic regret performance metric causes exprexxplora,oon and [2 proposes, predictive sampling, an algorithm that deprioritizes acquiring information that loses usefulness quickly.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Policy Gradient Algorithms for Stationary R..
38 presents the first finite time convergence of the
average reward two timescale Advantage Actor-Critic (A2C) to a stationary point. [4 further improved its rate by leveraging a single timescale algorithm. Convergence to global optima of A2C was analyzed in [5 36 which use a two loop structure with the inner loop critic cstimation. Further, 37 combines a mirror descent update with experience replay and characterized global convergence. Natural Policy Gradient (NPG) was analyzed in the discounted reward case BS 30 and with entropy regularization 440 NPG in the average reward setting with exact gradients was characterized in B.T. Most relevant to our work is the Natural Actor Critic (NAC) analyzed for the discounted reward case in 42 and average reward setting with (compatible) function approximation in 4
3 Problem Setting
Notation. Standard typeface (e.g., s) denote scalars and bold typeface (c.g., r, A) denote vectors and matrices. || lle denotes the infinity norm and ||2 denotes the 2-norm of vectors and matrices. Given two probability measures P and Q, (rv(P,Q) fx
Q(dx)) is the total variation distance between P(da) is the KL-divergence. For two sequences {an} and {b,}, an = (b,,) represents the existence of an absolute constant C such that on I Cb.. Further Ô is used to hide logarithmic factors. [S] denotes the cardinality of a set Given a positive integer T, [T] denotes the set {0, 1,2, -3.. 1}.
P and Q, while (kll)) = fx P(dx) log Q(dx]
3.1 Preliminaries: Stationary RL
Reinforcement learning tasks can be modeled as discrele-time Markov Decision Processes (MDPs). An MDP is represented as M = A,,, where S and A are, respectively, finite sets of states and actions, E IRl [A|x|S] is the transition probability matrix, with P(s'|s,)) a) E [0, 1], for s, E,S,a a A, and r RR [S|AA is the reward vector with individual entries {r(s, a)} bounded in magnitude by constant Un 0. An agent in state takes an action rT-ss) according to policy æ, where for each state s, T-|s)) is probability distribution over the action space. The agent then receives reward r(s, a) and transitions to the next state s' P(-ls, We denote the policy by E |[S||AI, which concatenates (T({)))). In a stationary MDP, the transition probabilities P and the rewards are lime-invarians
Markov Decision Process.
Average Reward and Value Functions. In this work, we consider the average reward sctting, essential in modeling problems where the importance of rewards does not decay with time g The time averaged reward of an ergodic Markov chain following policy æ converges
IT- r(s), a()
JR
lim T-000
Exndr PPagp.PPP... (1)ama((1))) [r(s,a)),
where d the stationary distribution over states induced by policy л and transition probabilities P. The relative stato-value function defines overall reward accumulated when starting from state as
Xe Σ
~ (*)s E
(r(s), at) - JJ) 0s =
where the expectation is over the trajectory rolled out by a ~ ((15)) and st+1 ~ P(-(s,,a)) Similarly, the relative state-action value function defines the overall reward accumulated by policy when starting from state and action a as
(~(saa):=
Σ
(1)) - |s = 8, 00 0,00=0 =
Natural Actor-Critie, The goal of an agent is find policy that maximizes the average reward
max JT max Ex E. (()aaa(()))) (r(s.
3

---

Published in Transactions on Machine Learning Research (02/2026)
Here, consider the actor-critic class of policy-based algorithms. While actor-only methods dasadvanlage due to inefficient use of samples and high variance and critic-only methods are a risk of the divergence from the optimal policy, actor-critic methods provide the best of both worlds [3 An actor-critic algorithm learns the policy and the value function simultancously by gradient methods. Further, the natural actor-critic leverages the second-order method of natural gradient to establish guarantees of global optimality The actor updates the policy no parameterized by by performing a natural gradient ascent step
0 - 0 3F Jo
where FR, Bo (aaamo.F))aaaaaaa ())amma(())))
v log (a(s))) (V log logls()))))
FR is called the Fisher Information matrix. The gradient of the average reward is given by the Policy Gradient Theorem 20 Section 13.2] as
V Band ( [QT a) V log To(a)s))
The crilic enables approximate policy gradient compulation by estimating the Q-Value function (*,, using TD-learning as
Q(s,a) t Q(s a) + + [r(s, a) - 1 Q(s',a)) Q(s,a)),
where 5 a)) of ~ x.16'), and yj is an estimate of the average reward J
3.2 Non-Stationary RL
In this work, we study reinforcement. learning with lime-varying encironments. The MDP is modeled by a sequence of environments M = (M, = (S, P,,re)) with time-varying rewards {rc} and transition probabilitics (P,). At cach time t, the agent state 8, takes action a,, receives reward (s, az), and transitions to the next state St+1 Po(-Ise, at). The cumulative change in the reward and transition probabilities is quantified in terms of variation budgets An,T and ArT
T-1
A RRT
Σ ||++1 - relloot APT = Σ |P - Prlloo:
+ APT.
(1)
Note that while the overall budgets AR,T, ArT may be used as inputs by the agent, the variations at a given time L, [| - relloo and [P++1 Pelloo are unknown.
We denote the long-term average reward obtained by following policy in the environment M, by
J = Pt (saat()))))))) [re a)].
Further, the state and state-action value functions at time solutions to the Bellman equations
= (s) = Σ (slls) and =((())))) - Ji +
E (()
REA
S'ES
The set of solutions to the Bellman equations above is Q (Q + Q E. R} where E is the subspace orthogonal to the all ones vector and Q is the unique solution in E 46
Dynamic Regret. The goal of the agent is to maximize the time-averaged reward Er- Tr(ss, /T. We measure performance using an equivalent metric called the dynamic regret defined as
T Σ
Dy.-Reg(M.TT) = E
J.
re(s), ax)
(2)
where πद = arg max, J is the optimal policy in the enviromment. M, (S, at time L. The optimal
average reward J associated with oद can be computed by solving the linear program (2 described in Section D.4 The model of change and notion of dynamic regret considered here are standard in the
4

---

Published in Transactions on Machine Learning Research (02/2026)
Note that it is more challenging to analyze and practical than static regret, which compares the cumulative reward collected by an agent against that of a single stationary optimal policy 48
non-stationary RL literature
12
Ar
18
16
17
19
Challenges due to Non-Stationarity. When running gaalcy-gradiet methods in stationary RL, the policy evolves efficiently learn a fixed environment. (P,r). However, in non-stationary case; the environment rt) also changes over time. Therefore, the agent chases moving target, namely, the time-varying optimal policy n resulting in the following unique challenges.
Explore-for- Chaange us Exploit: The agent needs to explore more aggressively than in the stationary sctting to adapt to the changing dynamics. As an example, a sub-optimal action at the current timestep may become optimal at a later timestep, necessitating re-exploration. This in sharp contrast to stationary RL, where sub-optimal actions are picked less often as time progresses.
Forgetting Old Environments: The policy and value function estimates must evolve quickly lest they might become irrelevant when the environment changes significantly. However, observations are noisy and the agent needs to collect multiple samples obtain confident estimates. Hence, an agent has carcfully balance the rale of lorgetting the old environment versus learning new one.
4 Algorithm: NS-NAC
In this section, we present Non-Stationary Natural Actor-Critic (NS-NAC), a two-timescale natural policy gradient method with a restart based exploration for change and step-sizes designed to carefully balance the rate of forgetting the old enviromment and adapting to new one. Note that we use the variation budget Ar as an imput to NS-NAC here. Since the variation budget may not always be known, we present a parameter-free algorithm BORI-NS-NAC in Section that does not require this.
Algorithm Non-Stationary Natural Actor-Critic (NS-NAC)
Input time horizon T; variation budgets ARTT, ApTs projection radius RQ Set step-sizes of actor 3, critic a, average reward as function of AR,T Artr number of restarts N of length H = [ as function of An,r Apri time t = 0; so some starting distribution
3: for n 0, ,,,,,
do
4: Set policy (a(a(s) = [ value function Qi(s, = Vs, a, average reward estimate π)- =
Take action ag T-(-
6: for h = 0, ,.. H do
7: Observe reward ri (8), next state 8+11 ~ Pe(:1st, at), take action a++1 ~ ny (-18814)1 M - 7l + (r, a1) - m1)
8:
Qc+1(se, az) f- IT Q1((t, ax) + (re(s), - m + Qe(s+11, - Qc(s)aa)))
Teti(als) x re(als) (xp())
10:
t+++1 Enee Te(a'(s) (xp()()))))(
Vs,
11:
12: end for
13:
end for
The NS-NAC algorithm seeks maximize the total reward received over the time horizon T, given the variation budgets AR, and APT. At timestep I, क. denotes the softmax parameterized tabular policy with parameters 0, R|S|| where (als)) 2 0 Va E A and E. (a(ss) Vs e S and represented
exp(0,)a.a
n: EvEA exp O,),
म = arg max JT is the optimal policy in the environment. Mr- The estimate of the tabular state-action value function Q denoted by Q [ISAAA) 7h denotes the estimate of the average reward J
NS-NAC divides total horizon T into N segments of length H [T/N] each. At the beginning of each segment, the hm restarts the NAC sub-routine (line 4 thereby ensuring that the a algorithm sufficiently
5

---

Published in Transactions on Machine Learning Research (02/2026)
explores for change. Next, at each time-step t= nH Vn E [M] E [H] the actor (slower timescale) takes natural gradient ascent step B8 towards optimal policy in environment M, as
<=(als) exp a)) T14i(als) ExeEs (a'a's exp (s, a'))
Dut1 t + QQT (agg
Vs,a.
In the absence of knowledge of the exact natural gradient, the actor uses an estimate of the value function to update the policy with tabular softmax parameterization as in line 10
The critic (faster timescale) estimates the tabular state-action value function of the current policy π. as Q: using TD-Tearning with step-size a (line 9h. The projection step in line is defined as Rng [x] == arg ly ||x - yll2 (see Lemma [ and following discussion on choice of RQ). Further, the average reward estimate ? is updated with step-size (line S Using a two timescale technique with a > B, NS-NAC thus enables the actor to chase the moving target केद facilitated by the critic updates of the value function estimates which adapt the changed data distribution. In the stationary RL case, this change in data distribution is induced solely by the evolving actor policy, while in non-stationary RL, the time-varying environment (P,, further exacerbates it. Further, as Theorem 1 suggests, careful selection of the step-sizes as function of the variation budgets enables NS-NAC to balance the rate of forgetting the old enwironment versus learning a new one.
Function Approximation. While we consider the tabular formulation here for the ease of presentation, NS-NAC can also be extended to the function approximation setting. Further details in Scction E
5
Regret Analysis: NS-NAC
In this section, we set up notation and assumptions and establish an upper bound on the dynamic regret of NS-NAC. We further present a sketch of the proof in Section
5.1
Assumptions
Notation. We denole an observation O₂ = (8t, at 1 0(+1). If P:t... is the stationary distribution induced over the states, we define the matrices A(O,), P. [11S||||((S||1 as
(st, 1, if (s), az) + (5141100-111 (8c, (8141, a(+1)
-1, if (st, az) # (s1+1,
A(O
0,
else
A P.
E Bend P (.). P(la [A as..a,s'.aa)
If D...P = diag (m P. (s)t((als)) and 1 is the all ones vector, then the TD limiting point
22 satisfics
Pt (I, J. + A =
(3)
A Markov chain generated by implementing policy and transition probabilities P is called uniformly ergodic, if there exists m > 0 and (0, such that
Assumption (Uniform Ergodicity,
3
drv (P(s, E -/s0 = s), d= < mp" VT 0
where P is the stationary distribution induced over the states. We assume Markov chains induced by all potential policies n, in all enviromments P., [T], are uniformly ergodic with parameters m, P. Further, if गł denotes the optimal policy for the environment M, = A, Pt, Pt),,), there exists C > 0 such that
C inf Pu (s)
a.f,l d= Pi (x) > 0.
Lemma 1 (Lemma..)) 2, 46). Under Assumplion 1. for all polential policies in all encironments Pt, l. (T), À P. is negative semi-definite
Define its macimum non-zero eigenwalue as -X.
Assumption standard in literature
Also note, we set the projection radius RQ = 2U 2URA in
line because (A..P.)) 1|2 --1 where t represents the pseudo-inverse.

---

Published in Transactions on Machine Learning Research (02/2026)
5.2 Bounds Regret
Theorem 1. If Assumplion 7 is salisfied, the slep-sizes are chosen as 0 < 0,,,7 < 1/2 and number of
restarts as 0 < N < T in Algorithm 1 then we have
T
Dyn-Reg(M.) = E
Σ t=
st re(s), or)
5 +
N
NT
+
37 +0 (TVB
cummlaatio change in policy over horizon T
effect of initialization
(4)
+ +
BT + + + (V NT
δ (TVa)
+0(
ArT
+
1//3//3
42/3
cummulativv change critic estimates
ErrOr in average reword estimate at critic
error due to non-stationarity
where AT = ART + APT, Ö(-) hides the constants and logarithmic dependence on the time horizon T. Choosing optimal x² = =-" (4)) 1/3 3 = 12 and N = A./TT// the resulting regret (with explicit dependence on the size of the catee-ccton space (1),1A| is
(M T) S Ô (151"31411/2///00777))
(5)
We provide a sketch of the proof in Section 7 and the full proof in Section Further, extension of NS-NAC to the function approcimation setting can be found in Section E
Effect of Non-Stationarity. The variation budget Ar represents the extent of non-stationarity of the environment. In Thcorem [ as the variation budget increases, so do the optimal choice of step-sizes and number of restarts, and the regret incurred (5). This observation is consistent. with the intuition that in a rapidly changing environment, the algorithm must adapt quickly and explore more (hence, larger step-sizes and more restarts). However, estimates, which soon become outdated (hence, higher regret). Also, in environments with larger state/action spaces, the agent requires proportionately more samples to detect changes and learn good policy.
result,, the algorithm cannot exploit its current policy and value-function
Remark. Define the mixing time 33 as Tiir == minfi 2 0/mpi~1 < min(a, 8)} with m, p as in Assumption 1 The explicit dependence of the upper bound, tracked in Section D on Taaz and À can be stated as
Dyn-Reg(M, T) Ô (18))/111)/2 Tiir 1/655/6
We choose not to include this explicit dependence in Theorem to maintain focus on the dependence of regret on the variation budget and prevent confusion by direct comparison to the diameter D of an MDP. Theorem 2 [16, Proposition 1). For any learning algorithm, there erists a non-stationary MDP such that the dynamic regret of the algorithm is at least 1/3 A 1////222 A 1/3/2/33 where D is the diameter.
Gap between Bounds. To the best of our knowledge, this is the first bound on dynamic regret for model-free policy-based algorithm in the infinite horizon average reward setting. Observe that the infinite horizon setting (only one sample per environment available) is harder than the episodic setting (environment remains stationary during the episode) and nocessitates a single loop algorithm with the policy being updatod every limestep. We conjecture that the gap between the bounds results from slack in the analysis of the underlying Natural Actor- Critic (NAC) algorithm. The best-known regret bounds for NAC' for infinite horizon stationary MDP in the (compatible) function approximation sctting with a two timescale algorithm is (T3/4) 42 Analysis of the actor involves the norm of the critic estimation error QQ Q (Proposition whereas guarantees for critic establish a bound on norm-squared of the error | - Q. (Proposition 2). This mismatch, which underlies the sub-optimality of the current best stationary infinite horizon NAC analysis, is exacerbated in non-stationary environments resulting the gap between the upper and the lower bounds. [ Note that. mismatch of the value function estimation error between actor and critic doesn't occur in the analysis of model-basel methods which use Hocffding style high probability bounds.
TTe term characterizing the difference in value functions at consecutive timesteps | Tet1 QT is the cause for the
bottlenock O (A//3Ta 1/3///3 + ) term (sec Ia, Is, Iu in Proposition 2

---

Published in Transactions on Machine Learning Research (02/2026)
6 Unknown Variation Budgets: BORL-NS-NAC
NS-NAC required knowlodge of the variation budget to achieve sub-lincar regret by choosing the optimal step-sizes and number of restarts as described Theorem | which not always possible in practice. To overcome this limitation, in this section, we propose a parameter- algorithm that adaptively learns the variation budget when it is unknown priori. Inspired by the bandit-over-RL (BORL) framework in T6 12 present BORL-NS-NAC that does not require prior knowledge of the variation budget Ar yet achieves sub-linear dynamic regret. Further, utilizing the EXP3.P analysis from
50
we present regret upper bound.
Algorithm Bandit-over-Rl RL Non-Stationary Natural Actor-Critie (BORI-NS-NAC)
1: Input time horizon T,, projection radius Ro
2: Initialize vo, = 0, Po.j [m Vj [[In 77, epoch length W
Set { 0.95 / In T
[l T7
/ IIm //T//T//T = 1.05
[1
[ [T/W]
for 0, 0, ..... [T/W] do
Sample Js ~ Ps where Ps,j = (1 ()
exp (Sur,r)
Σ, exp (Sme.s)
Tim T rJ
6: Set step-sizes Bg = (WT) 1/2
1/3
and restarts N, = W (TW/W 5/6
02 7/-
T
T
(++1)W11
Run NS-NAC (Algorithm Lt for W time-steps and observe cumulative reward Ra,j
Σ
re(Se, ac)
8: Update posterior as 1iti uaj
otI R.
Pi.5
end for
BORI-NS-NAC works by leveraging the adversarial bandit framework to lune the variation budget dependent. parameters, Le. step-sizes and number of restarts, in NS-NAC and hedges against changes in rewards and transition probabilities. Algorithm runs the EXP3.P algorithm 50 over [T/W] epochs with NS-NAC as sub-routine in each epoch. In each epoch, an arm of the bandit is pulled to choose the parameters of the sub-routine and the cumulative rewards received are used to update the posterior. The space of all possible parameters is discretized and the arms of the bandit are considered to be T (T",T"//lu")) /// T2/([n)) [ In each epoch i, arm ji pulled/samples from the distribution Pt (line as
,...,T.. T}.
exp (Eui.)) + [In T
exp
Ptj -1 -1-5) E, exp (Eus.))
(1
and step-sizes and number of restarts for the NS-NAC sub-routine as chosen as function of js described in line The cumulative reward Reg, observed in epoch is used to update the posterior (line) as
U441J = Uijj + Ij=j RaJ./W /
Pij
We now present an upper bound on the dynamic regret with the proof adapted from 16 12 which present parameter-free non-stationary model-free value-based and model-based algorithms respectively. While we defer the proof to Section we would like to highlight the additional cost of Ô ( WW
T
incurred by
w
rumning the EXP3.P algorithm to hedge against the unknown variation budgets and extra constant factor in the other terms. Note that for the optimal choice of parameters as discussed below, this additional regret
term continues to of the same order as the other terms δ (A /6/666 and hence BORL-NS-NE NAC continues to have the same order of regret NS-NAC.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Theorem 3. If Assumption 1 is satisfied, the time horizon T is divided into epochs of length W O(T2/3)
in Algoritlem then we have for any jt E {0, 10, [n]]]}
Dyn- M, o (m//nn
nn T.
+ O
TNt WBt
+ O
T W
NWW at
+
BTT af
+ O TV
(6)
W
of hedging EXP3.P
effect of inilialization
cumulativn change in policy over horizon
+0 STT 71
O T Tv>
+ +
NEW
Ar NI
+ +
A T2/33 Val
A 7//3
W
cummlation change in critic estimates
ervor it overage reward estimate at critic
error due to non-etationerity
where = + APT, al =1 hides constants and logarithmic dependence on lime horizon T. Choosing optimal value of and resulling 1/2 and N = A4 4/6/1/66 we upper bound the regret as
(207/ (
1/3
,Bt
(ot//luT] 1/2
-N W (2 (201/1 /1uT) 3/6
and Ö.)
T
optimal poramelers as - 1/3
,
-Ap)
Dyn-Reg(M, M, s SO
7 Proof Sketch of Theorem 1
We now present sketch of the proof of Theorem that presents upper bound regret of NS-NAC and address the following theoretical challenges that non-stationarity poses. (a) Stationary environment NAC analyses use the KT-divergence to the optimal policy as a Lyapunov function. What is an appropriate function for dynamic environments where the optimal policy varies with time? (b) How do the simultaneously varying environment and evolving policy alfect the estimation of the average reward and state-action value function? (c) How do the time-varying transition probabilities affect the martingale- based argument used analyze the Markovian noise?
We start by decomposing Regret Decomposition.
Dyn-Reg(
E
J
- - ( a:)
(8)
L difference actual vereus instantaneous
difference of optimal versus actual average reward
1
where 1 measures the performance difference between the average reward of the actual policy æ, at time relative to the optimal policy πミ The second term In analyzes the gap between the average reward and the actual rewards received due to the stochasticity of the Markovian sampling process.
Actor (Proposition 1). We first bound I in 8 by adapting the Natural Policy Gradient analysis for average-reward stationary MDPs in 41 to non-slationary environments. NPG in the stationary case is analyzed by characterizing the drift of the policy towards the optimal policy using an appropriate Lyapunov function. non-stationary case we innovatively separate out and analyze the change in the environment from the drift of the policy as follows. We start by dividing the total horizon T into N restarted segments of length H each and split J
N-1
R
Σ
Tatt JT
TT An
(JRnHth
J aailh aH+h
Js: optimal avg. reward across two enviromments
avg. reward ldaub-optimality
reward with same Is polpolyyy environments
We benchmark policies learned in each segment n E [N] against the optimal average reward at the initial timestep nH i.e. f We bound I4 by mirror descent style analysis for each segment with &

---

Published in Transactions on Machine Learning Research (02/2020)
{nH...., 1)H - 1} by the Lyapunov function adapted to non-stationarity
W(T:) Ed Pan (
In addition, since NS-NAC does not have access to the exact value functions Q la also depends on the critic estimation error | Q:llaa.
We analyze the change in the environment next. We bound 13, the difference optimal average rewards in two different environments, in terms of corresponding changes in the environment || TwH and P Pn (Lemmna 5 by clever use of the linear programming formulation of an MDP. Similarly, deftly bound Is, the difference in average rewards when following the same policy TH in two different environments, in terms of the environment change (Lemma 0 The number of restarts N balances exploration-for-channg and learning a good policy and we optimize it in Theorem to minimize regret.
We bound the critic cstimation error s, = HE [Q, Q, p for each restarted segment n E [N] where = {nH..... (n + 1)H - }} by adapting the critic analysis used in stationary MDPs 46 to non-stationary environments. Tf O = (8t, 0t 81+1, , we can decompose the error as
Critic (Proposition 2).
|| 12 1 (1 (1-a)))lbbll a a [(r.(O.) - (O,) + A(O))T)) + (A(O.) A R. Ub!
Isverror due Markov Doise
+ a (P..))nn(O.))) Iy:vv, reward ostimation error Isivalue function drift
[1 Q
T+ 112 a² |I,) - n (
(9)
Igevariance term
Is is the error induced by the Markovian noise which is analyzed leveraging the auxiliary Markov chain described below. I7 describes the error due to an inaccurate estimation of the average reward which is bounded below. Is, the change in the true value function is caused by drifting policies and environments, and can be neatly bounded in terms of the change in policy, rewards and transition probabilitics (Lemma 9 Finally, Jo is the variance term.
Bound on Markovian Noise. For each restarted segment n E [N], consider time indices (n (1H1)) > nH. Consider the auciliary Markoe chain starting from 8t-r constructed by conditioning on Fi = (s -1-1-1, P.-r] and rolling out by applying ---r-1 P.-
P
me
3 84
We
āz
P
8,41
mec 2 0111-
Recall that the original Markov chain
Pt-r, 86-r+1 06-7+1 13 8t Te-1, az Ps, s1+1
Te a++1-
8t-
at-7
This enables us to characterize properties of original Markov chain compared to the auxiliary chain as drv (P(O, E -FFr)), P(O, E (Frr)) where Ô, = (3,, āz, 5t+1, d.+1). We do this by bounding the effects of drifting policies and transition probabilities in the original chain and leveraging uniform ergodicity in the auxiliary chain. While prior works use auxiliary Markov chains for stationary environments is the first adaptation to a non-stationary enviromment. Observe that. the time-varying transition pilbabilities P. add extra layer of complexity, unlike the stationary case where only the policy changes over time.
g ours
Average Reward Estimation Error (Proposition 4 To bound 1 in 9 i.e., the error in the average
reward estimate & 1) J we can decompose the error as
o S (1 7) +
((:(0,))
- + (r:(0
Jaoterror due to Markov nnise
ang avg
consecutive timestops
Jazivariance term
2T g [x] arg minyce / ||x ylla the projection E, the subspace orthogonal the vector
10

---

Published in Transactions on Machine Learning Research (02/2026)
30003-
VS-NAC
1.8
DORL NAC SW-UCRLSCCV
BOAL NAC SW-DCRLI-CCW
Dyuanic Regret
Dynamic Regret
Dyuanic Regret
x 101
Var UCILL2
Var UCRL2
30.03-
RestartQ UCB
3 10"
RestartQq UCB
NS-NAC
1.2x 10
BORL-NS-XAC
1000
SW-DCRLa-WW
Var DCRL2
Resturt UCB
0.0
1.0
25
10
10"
10
Time Step
T
Ar
(a) 18
50,
[4 4, Ar 300
(b) [S] 50, |A| = 4, Ar 300
(c) |6| = 50,
|A| 5 10a
Figure 1: Performance of NS-NAC and baseline algorithms across various settings. (a) Dynamic regret for
single instance with = 25 x 10a steps. Log-log plots showing the effect of varying: (b) time horizon T, and (c) variation budget Ar.
Jo is analyzed using the auxiliary Markov chain construction. In quantifies the difference in average rewards at consecutive timesteps, and is neatly bounded in Lemma @ in terms the correspouding changes in policies, rewards, and transition probabilities. 412 again the variance term.
Finally, 12 in (8) characterizes the difference between the average reward and the instantancous reward at any time, and is analyzed in Proposition B using the auxiliary Markov chain to bound the bias occurring due to Markovian sampling. This concludes the proof sketch.
8 Simulations
We empirically evaluate the performance of our algorithms synthetic non-stationary MDP (see Section G comparing it with three baseline algorithms: SW-UCRL2-CW 5I1, Var-UCRL2 [3 and Restart UCB 16. SW-UCRL2-CW is a model-based algorithm that adapts to non-stationarity by maintaining a sliding window of recent observations, applying extended value iteration, and adjusting confidence intervals to track changing dynamics. Var-UCRL2, also observed variations in rewards and transitions. Mlodel-free RestartQ-UCCB periodically restarts Q-learning and resets its upper confidence bounds to adapt to non-stationarity. While there is gap between our theoretical regret analysis and those of the baseline methods, empirically observe in Figure [ that NS-NAC and BORL-NS-NAC strongly match their performance achieving sub-linear dynamic regret across all settings. Further, we observe in Figure T(b) and Figure 1(c) the sub-linear effect of varying time-horizon T and variation budget Ar on the dynamic regret. See Scction for more experimental analysis.
adjusts its confidence intervals dynamically based on the
9 Conclusion
We consider the problem non-stationary reinforcement lcarning in the infinite-horizon average-reward setting and model it as an MDP with time-varying rewards and transition probabilities. We propose and analyze the first model-free policy-based algorithm, Non-Stationary Natural Actor-Critic. A two-timescale natural policy gradient based method, NS-NAC utilizes restarts to explore for change and learning rates as adapting factors to balance forgetting old and learning new environments. Further, present bandit-over-Rl based parameter-free algorithm BORL-NS-NAL NAC that does not require prior knowledge of the variation budget and adaptively tunes step-sizes and number of restarts. Both algorithms achieve sub-linear dynamic regret, thus, theoretically validating ing policy gradient methods often used in practice in continual non-stationary RL.
11

---

Published in Transactions on Machine Learning Research (02/2026)
References
[1] Christopher Ych, Victor Li, Rajeev Datta, Julio Arroyo, Nicolas Christianson, Chi Zhang, Yize Chen, Mohammad Mehdi Hosseini, Azarang Golmohammadi, Yuanyuan Shi, et al. Sustaingym: Reinforcement learning environments for sustainable energy systems. Neural Information Processing Systems, 2024.
[2] Shi-Yong Chen, Yang Yu, Qing Da, Jun Tan, Hai-Kuan Huang, and Hai-Hong Tang. Stabilizing reinforcement learning in dynamic environment with application to online recommendation. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018
[3] Yash Chandak, Georgios Theocharous, Shiv Shankar, Martha White, Sridhar Mahadevan, and Philip Thomas. Optimizing for the future in non-stationary mdps. In International Conference on Machine Learning, 2020.
[4] Yash Kanoria and Pengyu Qian. Blind dynamic resource allocation in closed networks via mirror backpressure. Management Science, 2024.
[5] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, 2021.
[6] Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement lcarning. arXiv preprint arXiv:2006.005. 05826,
2020.
[7] Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes, worst-case approach using model-based reinforcement learning. Newral Information Processing Systems, 2019.
[8] Sridhar Mahadevan. Average reward reinforcement learning: Foundations, algorithms, and empirical results. Machine learning, 1996.
[9] Jan Peters, Sethu Vijayakumar, and Stefan Schaal. Reinforcement learning for humanoid robotics. In International Conference on Humanoid Robots, 2003.
[10] Neharika Jali, Guannan Qu, Weina Wang, and Gauri Joshi. Efficient reinforcement learning for routing jobs in heterogeneous queueing systems. In International Conference on Artificial Intelligence and Statistics, 2024.
[11] Bai Liu, Qiaomin Xie, and Eytan Modiano. Rl-qn: A reinforcement learning framework for optimal control of queueing systems. ACM Transactions Modeling and Performance Evaluation of Computing Systems, 2022.
[12] Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. In International Conference on Machine Learning, 2020.
[13] Ronald Ortner, Pratik Gajane, and Peter Auer. Variational regret. bounds for reinforcement learning. In Uncertainty in Artificial Intelligence, 2020.
[14] Pratik Gajane, Ronald Ortner, and Peter Auer. sliding-window algorithm for markov decision processes with arbitrarily changing rewards and transitions, 2018.
155 Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 2010.
16) Weichao Mao, Kaiqing Zhang, Ruíhao Zhu, David Simchi-Levi, and Tamer Basar. Model-free nonstation- ary reinforcement learning: Ncar-optimal regret and applications in multiagent reinforcement learning and inventory control. In Management Science, 2024.
17] Songtao Feng, Ming Yin, Ruiquan Huang, Yu-Xiang Wang, Jing Yang, and Yingbin Liang. Non-stationary reinforcement learning under general function approximation. In International Conference on Machine Learning, 2023.
12

---

Published in Transactions on Machine Learning Research (02/2026)
[18] Huozhi Zhou, Jinglin Chen, Lav R Varshney, and Ashish Jagmohan. Nonstationary reinforcement learning with linear function approximation. Xiv preprint arXim:2010.02444, 2020.
[19] Ahmed Touati and Pascal Vincent. Efficientt learning in non-stationary linear markov decision processes. arxiv preprint or 0rXiv:2010.12870 2020.
[20] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[21]
Sindhu Padakandla. A survey of reinforcement learning algorithms for dynamically varying environments.
ACM Compuling Surveys (CSUR), 2021.
[22] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement
learning: A review and perspectives. Journal of Artificial Intelligence Research, 2022.
[23] Hyunin Lee, Ming Jin, Javad Lavaci, and Somaych Sojoudi. Pausing policy learning in non-stationary reinforcement learning. In International Conference on Machine Learning, 2024.
[24] Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Conference on learning theory, 2021.
[25] Binghui Peng and Christos Papadimitriou. The complexity of non-stationary reinforcement learning. Tn International Conference on Algorithmic Learning Theory, 2024.
[26] Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for non-stationary bandit. problems. arXiv preprint arXiv:0505.3415, 2008.
[27] Omar Besbes, Yonatan Gur, and Assal Zeevi. Stochastic multi-armed-bandi problem with non-stationary rewards. Neural Information rocessing systems, 27, 2014.
[28] Vishnu Raj and Sheetal Kalyani. Taming non-stationary bandits: A bayesian approach. arXio preprint Xiv: MXXi::100.000 09727, 2017.
[29] Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Learning to optimize under non-stationarity. In International Conference Artificiol Intelligence and Statistics, 2019.
[30] Su Jia, Qian Xie, Nathan Kallus, and Peter I Frazier. Smooth non-stationary bandits. In International Conference on Machine Learning, 2023.
[31] Yueyang Liu, Xu Kuang, and Benjamin Van Roy. A definition of non-stationary bandits. arXiv preprint or Xrxxi22302.111 12202, 2023.
[32] Yueyang Liu, Benjamin Van Roy, and Kuang Xu. Nonstationary bandit learning via predictive sampling. In International Conference Artificial Intelligence and Statistics, 2023.
[33] Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Newral Information Processing Systems, 2020.
[34] Xuyang Chen and Lin Zhao. Finite-time analysis of single-timescale actor-critic. Neural Information Processing Systems, 2023.
[35] Qinbo Bai, Washim Uddin Mondal, and Vaneet Aggarwal. Regret analysis of policy gradient algorithm for infinite horizon average reward markov decision processes. In AAAI Conference on Artificial Intelligence, 2024.
36] Yashaswini Murthy, Mehrdad Moharrami, and R Srikant. Performance bounds for policy-based average reward reinforcement learning algorithms. Neural Information Processing Systems, 2023.
[37] Nevena Lazic, Dong Yin, Yasin Abbasi- Yadkori, and Csaba Szepesvari. Improved regret bound and experience replay in regularized policy iteration. In International Conference on Machine Learning, 2021.
13

---

Published in Transactions on Machine Learning Research (02/2026)
[38] Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: optimality, approximation, and distribution shift. Journal of Machine Learning Research, 2021.
[39] Sajad Khodadadian, Prakirt Raj Jhunjhunwala, Sushil Mahavir Varma, and Siva Theja Maguluri. On the linear convergence of natural policy gradient algorithm. In IEEE Conference on Decision and Control, 2021.
[40] Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 2022.
[41] Yashaswini Murthy and R. Srikant. On the convergence of natural policy gradient and mirror descent-like policy methods for average-reward mdps. In 2023 62nd IEEE Conference on Decision and Control (CDC), 2023.
[42] Sajad Khodadadian, Thinh T Doan, Justin Romberg, and Siva Theja Maguluri. Finito-sample analysis of two-tine-scale natural actor critic algorithm. IEEE Transactions Aulomalic Control, 2022.
[43] Yudan Wang, Yue Wang, Yi Zhou, and Shaofeng Zou. Non-asymptotic analysis for single-lopp (Natural) actor-critic with compatible function approximation. In International Conference on Machine Learning, 2024.
[44] Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica, 2009.
[45] James Martens. New insights and perspectives the natural gradient method. Journal of Machine Learning Research, 2020.
[46] Sheng Zhang, Zhe Zhang, and Siva Theja Maguluri. Finite sample analysis of average-reward learning and q-learning. In Neural Information Processing Systems, 2021.
[77] Yingje Fei, Zhuoran Yang, Zhaoran Wang, and Qiaomin Xie. Dynamic regret of policy optimization in non-stationary environments Neural Information Processing Systems, 2020.
[48] Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Online markov decision processes. Mathematics of Operations Research, 2009.
[49] Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function approximation. Neural Information Processing Systems, 2019.
[50] Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 2012.
[11] Wang Chi Cheung, David Simchi-Lovi, and Ruihao Zhu. Nonstationary reinforcement learning: The blessing of (more) optimism. Management Science, 2023.
[22] Ofrr Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Neural Information Processing Systems, 2017.
[33] Omar Domingues, Pierre Menard, Matteo Pirotta, Emilie Kaufmann, and Michal Valko. A kernel-based approach to non-stationary reinforcement learning in metric spaces. In International Conference Artificial Intelligence and Statistics, 2021.
[4]]
Amir Bock. First-order methods in optimization. SIAM, 2017.
[55]
Martin L. Puterman. Markoe Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
[66] A Yu Mitrophanov. Sensitivity and convergence of uniformly ergodic markov chains. Journal of Applied Probability, 2005.
14

---

Published in Transactions on Machine Learning Research (02/2026)
[57] Long-Fei Li, Peng Zhao, and Zhi-Hua Zhou. Dynamic regret of adversarial linear mixture mdps. Advances in Neural Informalion. Processing Systems, 2023.
[58] Peng Zhao, Long-Fot Fei Li, and thi-Hua Zhou. Dynamic regret of online markov decision processes. In International Conference Machine Learning, 2022.
[59] Yingying Li, Aoxiao Zhong, Guannan Qu, and Na Li. Online markov decision processes with time-varying transition probabilities and rewards. ICML workshop on Real-world Sequential Decision Making, volume 3, 2019.
[60] Carl D. Meyer, Jr. The condition of a finite markov chain and perturbation bounds for the limiting probabilities. STAM Journal on Algebraic Discrele Melhods, 1980.
[61] Dimitri Bertsekas. Dynamic programming and optimal control: Volume I, volume 4. Athena scientific, 2012.
62] Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Neural Information Processing Systems, 1999.
633 Vijay R. Konda and John N. Tsitsiklis. On actor-critic algorithms. SIAM Journal on Control and Optimization, 2003.
15

---

Published in Transactions on Machine Learning Research (02/2026)
10 Appendix
Contents
Introduction
1
2
Related Work
2
3
Problem Setting
3
3.1
Preliminaries: Stationary RL
3
3.2 Non-Stationary RL
4
4
Algorithm: NS-NAC
5
5
Regret Analysis: NS-NAC
6
5.1 Assumptions
6
5.2 Bounds on Regret
6
Unknown Variation Budgets: BORL-NS-NAC
8
Proof Sketch of Theorem
9
8
Simulations
11
Conclusion
11
10 Appendix
16
Additional Related Work
18
B Notation
19
Symbol Reference
20
D Regret Analysis: NS-NAC
21
Actor
D.2 Critic
24
D.3
Average Reward Estimation
27
0.4
Technical Lemmas
30
D.4.1 Actor
30
D.4.2 Critic
33
D.4.3 Average Reward Estimation
36
D.5
Auxiliary Lemmas
36
16

---

Published in Transactions on Machine Learning Research (02/2026)
D.5.
Actor
36
D.5.2
Critic
37
D.5.3
Average Reward Estimation
39
D.6
Preliminary Lemmas
41
D.7 Universal Dynamic Regret
43
E NS-NAC with Function Approximation
44
F Regret Analysis: BORL-NS-NAC
45
G Simulations
47
17

---

Published in Transactions on Machine Learning Research (02/2026)
A Additional Related Work
Table 1: Regret comparison across Non-Stationary and Stationary RL algorithms with variation budget Ar, time horizon T, episode length H, size of the state-action space [S], [Al, maximum diameter of MDP D, dimension of feature space d and dynamic Bellman Eluder dimension d.
Model
Policy
Setting
Algorithm
Regret
Free
Based
Lower Bound
(15 | D (
|SS|AA] DL
Non-Stationary
Infinite Horizon
X
[18)8|A41 DA
Average Reward
X
Ô (ATT
6 ((5) LA T
This Work
Lower Bound
2 |||
0
AAT's)
Non-Stationary
24
Episodic
dt H
LAi A
16
d$ d
Non-Stationary
18
X
di ddAAHiTi
Episodic Linear MDP
x
Stationary
Infinite Horizon
22
Discounted Reward
Stationary
Infinite Horizon
Average Reward
18

---

Published in Transactions on Machine Learning Research (02/2026)
B
Notation
Variation Budgets
RRT
|r411 - -elloc:
|r,
r 001
1=0
1=--++1
T-1
PPT
P141
=
Σ P
1=0
AT = ART + APT.
The critic update (line g in Algorithm 1 can be defined in vector form using the following notation. Note that one-to-one mapping f A 21. | to map state-action pairs (s, S A to vector/ / matrix entries. However, for ease of notation, we denote the index of each entry by (s, a), instead of the more accurate fs),a).
O (8t, dt St+,1 0c+1)
(O, = [0; ... (8 a:); ();00 0; ... ;0' E IISIIA
(O+)) [0: ... :0: 0; :0] RISSIAI
(O,) -0;-..: ;7; 0 E IAAIA
A(O) E [ISAAAIIISI such that
{
if (s. = (s,
A(O);; = A(3,0,8,))ss
if (s. (s',a'), (s.a).j = (8',a') else
As result, we get the critic update
Q+1 = =I[ + a (r, - (0.) + A(0))
R2
For the purpose of analysis, we define the following quantities.
A P = P()
(s,,,,,8','
TT. =Q associated with n, P.P
JT. = Σ P. (s) Σ ((als)r(s,a)
[g = arg min ||x - ylla where E is the subspace orthogonal to the all ones vector 1 YEE
(Error in the functionniion estimate) T(π, P,r, r, 1. 0) = /² (r(O) FT....r P A(O) + + (A A ))
4. 1g [Q, 0
2b
Φt = 1tt - J
(Error in the average reward estimate)
N, 0) = (1 - J PT - (T)
Given time indices t
0, consider the auxiliary Markoe chain starting from St-r constructed by
conditioning on St-T, - P.- and rolling out by applying -i Pt- as
8t-7 Ot 84- -i--- ā--441
P ....I 5, - ā. 88t1
P
Wt-r-1 āe+1-
Recall that the original Markov chain
St-r ht-r- Pi-r
Pi-r+i
Ti-1
P. S1+1
me a+11
at-r 81-141 01-r41
...8,
19

---

Published in Transactions on Machine Learning Research (02/2026)
C
Symbol Reference
Constant
First Appearance
Ur
Section 3.1
UQ
Lemma 24
N, H
Algorithm
C= inf (s))
Assumption
s,,, x Ps()
m. P
Assumption
M [log, m X
Lemma 22
I-p
Lemma
W1 (3G)) 1/3 (AU
Proposition
19
W2 = (307) (400))))
Proposition
D1 = L2 + AURV / + AUR
Proposition
9
D2 = RU'R + Lp
Proposition
9
D3 = AURF + SU?
Proposition
D1 = 932
Proposition
W3 = (AU))
Proposition
W4 = (A4R))
Proposition
Bi = /AUB
Lemma
B₂ = UQ
Lemma
B3 = (Fim + G + Fs |S||A| + F2)
Lemma
B4 = F2 n + 2Uq)
Lemma
B5 = F2Gn
Lemma
Bg = FIP + F2Gp + F3
Lemma
By = (FiLm + Fr Prv[S|AA + F2)
Lemma
Bs F7 + F&Lp
Lemma
L AUR(M + 1) AVSSIA
Lemma
Lp = AURM
Lemma
G 2 /
Lemma
Gn 2X 1 VS|AA
Lemma
Gp = (X- 40aA M M
2 11)
Lemma
Fix = -U + 4UqG,
2)|S|AAI
Lemma
Fir = 2Uq 1. + UUqGr WQGG++U3(00 +1)
Lemma
F2 = 2U 180Q
Lemma
F3 16UnUq + 2402 S
Lemma
F. = SURUQ + 24U3 |
Lemma
16
F/ 4UR
Lemma
FG 2UR R
Lemma
F7 SU2
Lemma
20
F& = SU?
Lemma
21
20

---

Published in Transactions on Machine Learning Research (02/2026)
D Regret Analysis: NS-NAC
Theorem 4. If Assumption i is satisfied and the step-sizes are chosen as 0 < 0, 1/2 and number of then we have
restarts as 0 < N < T in Algorithm 1
r
Dyn-Reg( =
Σ J ae)
s so N Ô ( {0(m))
NT
+ Ō
T 3
cumulation change in policy over horizon T
effect of initialiiattio
+ RT +0 + (V
(TVa) +0( ((A + +
T2/3 A 1//3
curmulative change in crêtic estimates
error in anerage reward estimate at critic
crvor due to non-stationarity
where AT ArT AP,,, (-) hides the constants and logarithmic dependence on the time horizon T.
Choosing optimal a² - ( 1/3 3 = - ( and N A//T110 the resulting regret (with explicit 1/2
dependence on the size of the state-action space [|)) is
Ryn-Reg( S 2 (111"9144/"4//""351)
Proof. Recall that Algorithm
divides the total time horizon into N segments length H = [
T
T--
1
J
re(s), at)
=E
m
+ E
J
t=0
N-1 #-1
T
+ θ
N 3
+ + (BT) +2
Σ
E
|
1 - QnH [loo
N
==0
h=
E
J,m
as)
N-1
H-1
1/2
ArT
N +0 +0( (BT) +2 Σ
H2/2
Σ
E
[g Q QnH+a] 11
N
T
+ E
Σ
J: - r2 ar)
N-1 Σ
0
ATT
3 0 (BT) +2
④ (VII) +0
δ (Vol)) +0 8111
(
N
==
(VTH) θ (111) +
1H H2/3
1/3
+ 0
(/HH)
va
+0
+E Σ J a))
0 (ArT) N
( ( + + +
0 N
+0 (T + + 8T
A 2//31//3
T + + (1 NT
+
V
11/372/3
+0
+ 0 (ArT),
21

---

Published in Transactions on Machine Learning Research (02/2026)
where (a) is due to Proposition 1 (6) is by Proposition 2 and (nH((+)) ( (p.aH,(ttt)
(c) is by Jensen's inequality, År ART + Art and Proposition 3 We further have Olog 7). Note that (-)) hides constants and logarithmic terms.
D.1 Actor
The next result bounds the performance difference, measured by the average reward, between the optimal policies मद and the current policy πe.
Proposition 1. If Assumption 1 holds, we have
- TART, N
(U&+
Lp
TAPT N
E
Σ 1=0
J
J
2 2G
Lp 2Gp
C
error due to non-stationaritg
N
H
log A B
BTT
Un constant
E
[1
HH QnH+h 1l
+
h=0
x bias of imitialization
coumadatioe change in policy
critic estimation error
T-1
where RRT Σ
|rr+] rello, APT 1 Pello,, C is defined in Assumplion T is the tolal
t=
1=0
time horizon and N is the number of restarts. The remaining constants are defined Section C
Proof. Recall that Algorithm divides the total time horizon T into N segments length = J
each segment (indexed by a [N],, we use J as an anchor against which compare the performance of the learned policies.
N-16 #-1
F - s + (J JmHHin + (JHHII J nHHhh
E
E
h=0
(a)
-
ran 1ox (UQ + Lp)[[Pnntk Pamllloo++ (Jrin" Jm
N-1H-1
Σ
H (2 InH+h-1llo + (Uq + [P]][PHHH Pauth-alle))+ + (JTh" J juuutth
R=0 h=0
N H
H (2ART + (Uq+ LP) +
Σ
J
(10)
n=0
h=O
where (a) is by Lemmal and Lemma and (b) is by triangle inequality. We now bound the last term as
aatth
nH
N 11111
Σ
ΣΣ
Tist Pa [BQ (s,a) - 3V T. + (s)]
n= h=0
M (a|s)
(3QTH (
- WWaHh (QuH+4(ssa) 3QnHI+(ssa)
n.k.
d PuH (5)TT TH (a|s)
[3 - 3Q (s. + BVTHHt (s) - (s)
n.haa.a
d P Pal (8) t (a|s)
Tm (s, a)
BVTTHIA (s) (Q.uth(s,a) (Q.HH+
n.k.s.a
22

---

Published in Transactions on Machine Learning Research (02/2026)
2 Q th
Oulth lboo
(d)
d P.H (S) nH (a)s)
[3Q (s, -
+ BQNH+A(s,a) 8QnH+n(s,a))
n,h,s,a
--1
2G - Tamthllo + +GGp | PnH - P Pnittillo
(
d NT Pall (s)T))(((s))
log ZnH)th(s) 3V Wh MaRH--A
nH
h
d nn nt (8) ni (a|s)
Tnll i (als) Tn + #.QHHHh (s, - BQ BQnH+
log
B
D,
12
+H
+ 2GpAp.))
(11)
where (c) follows from the Performance Difference Lemma (d) follows from Lemma 8 and (e) from the actor update equation (line 10 in Algorithm and Zı(s) Lo'ed (a(ls)) exp(3Qe(s. as)))) Next, we bound each of 1, 12, Iz. Using Lemma 2
have
log (h))
11 ΣΣΣ
d Pan (s)
V V (s)
Σ
The (als)
3
J all+h+11
Bi9
Qni+tille +
C
C
Fm - Fuutalls + PPHPH4441 PaH
(12)
Next, establish bound on I2 as
ΣE
1
d Park (8) (s) log TnH+s+1(a(s)
B
Σ
Σ
Tu P.H (8) [DKL (s)) DEL. (T -n
8 n=0
1
T P.H (s) [KL (T
DKL
3
dh Fin Pun Ds) ( (+I))[[T
3
(
A
N Nlog |A|
B HH Panl (s) log
<
(13)
B
where (f) is because of non-ncgativity of KL-divergence and (g) is due to the restart in line of Algorithm 1
Lastly, 13 can be bounded as
= ΣΣΣΣ
Th P.H (s) Tom+h (s, a) QnH+n(s,a))
ΣΣ
Quif-
THHtk
Qutt-nllo-
(14)
We substitute the bounds on J1 Iz, Ia from set of solutions to the Bellman equations is
14 in 11 and then combine with (10). Recall that the
QQ ell Q E, E RR where E. the subspace
23

---

Published in Transactions on Machine Learning Research (02/2026)
orthogonal to the all ones vector and QTe is the unique solution in E 46 Finally, we use the equivalence |0r Qrl1 ||T [Qr Qr) loc. to get the result.
D.2 Critic
In this section, characterize the error in the critic estimation.
Proposition 2. For any [N], if Assumplion 7 is salisfied and 0 < < 1/2, then we have
nH -H+H-1
Σ
E [
QQ: - 111
A 2/3
H133
A
2/3 P.H.H (n- 1)H
H13
+0
R.nH 1)H
+
a
effect initialization
error due non-stationarity
H1/3 Rn 1)
2/3
A
2/3
(nHH(n++)FF H1/3
1
① (+ +0
+
+ o
+
7
72
7
7
in average reward estimate at critic
B a²
+O (ST) + δ
(all)
cumuulative change estimates
cunsulative chanae in policy over harizon T
where O(-) hides constants and logarithmic terms which be found in Equation 17 and ARnH()41)
E (n+1)) [|r+1 relloo, and A (atl(() = (1+1) |P Pelloo.
Proof. Recall that 2h = Ig Q2 0, E is the subspace orthogonal to the all ones vector 1 and the critic update equation (line in Algorithm D can be expressed vector form as Q = UR [Q, + a (O,) - (O,) (O,) Q:)]. Recall the notations rt, Tr, A(O)), Am.P: J,,(,,,, Φe from
Section We therefore have
1 12 |[ QQ+1 2 11
< || [Q + a (r,(O)) (O.)) ((O)) TH 113
I [sb, + (r.(.)) - m((O,) + Q())) + QT Q.1
< 11 111221000 + 2 (r((O:)mm(C0) (0.)) a(0))
+ 2 11 [Q - - Q"11' 2a (0,)) A(O.) 11 [Q Q
< 11 11 + 200h (r.(O.) ((00))))(..)))) A(O,) - A T. P. + 20r// A P. sb.
2 HI Q
(0,) A(O,) 12 2
Q T 11
< 11 + 2 (r.(0:) (O,) + +A()) - 20 (A(O.) T.
wb + 2a A P..Pi
2 11 Q7 Q 20²1|1;(0)) (0.))) AO,)). + 1
Q Q T
< 11 + (TT P,, //, 0,) + 200T 7 (0,)) 200h T. ST
+ 2 Hg [Q. -Q1+11 + 22 (r, (0,) + A(0)), 2 [Q
Q
2++
(4)
< 11 + 2o1' 2 Pt, Fr, vh, 0,) 20 v 112 (O.) (0.)), 2006 P..PPihh
+ 1| Q Q ||2 + 20 | (O.) - (0.) A(O, Q + 2
Tr+1
QT
Q
Tes1
< || (M, Pt, Tı, 1,, 0,) + 2n R || me - 2n 1
2| sb ||2 TT [Q. Q1111 ||2 20 11 -m.(..)
A Q+ T QT TT
24

---

Published in Transactions on Machine Learning Research (02/2026)
1 < 2a + 2aI
Pt, rr 1h 0,) 2/11/11 7/c
2/ 122 وg Q T Q,41 ||2 + 2a (2UR + Q + 2 Q - l1
- (1 2cA) /
1 + 2xT ,T,, Pt. To 12 0,) 20: 20 1 T/. + 2 2
[
[
2
lle
+ 2|| bb 22 E [Q Q241'] lla 2a (2UR ²² + + [ Q411 12
where (a) due to Cauchy-Schwarz inequality, (b) follows from w. E E and Lemma 1
Taking expectation, rearranging the terms, setting TH min ( 2 0/mp'~1 S over time, we have
and summing
-
Σ (=n
AE
[
-H+H-1
||
nH+HH1
nH+H-1
Σ
Σ
( Pe. 4r. Oil + Σ
|
2a
fm
TH
TT
12
nH+H-1
E
1b 11a [T [Q_
nH H
1/. 12
Q44
Q
a
(=n +T
Inf
nH+
E
[1
+ a2UURR +2 -T))
Σ
1-1
(15)
We now bound each the terms starting with the first term as
1 B nt
12
[2]
20
S
20
a
By Lemma 10
we have
nH H
I2 S
Σ
Ba + 1)2 + Bạ0TH RBARR-MMHTHtH BITHARA-nt--tttl
t= +TH
< BrB(rn + +1) -T)) BA TH) +
+ BB B (
By the Cauchy-Schwar Schwarz inequality, we have
nH -
n
1/2
n
1/2
13 <
Σ
V E B
Σ
E[oi]
Σ
Bi1l
t n +H
where Σ
B[d can be further bounded using Proposition
Using Lemma
we have
2U@ n
2
nH+H-1
1 5
Σ
Q.
The
12 <
Σ
GRllTE-1 - + PP41 - -Peloo
2
< (GRARnHInNDD Gp (mA
25

---

Published in Transactions on Machine Learning Research (02/2026)
Using the Cauchy-Seiin Shuchy-Schwarz inequality and Lemma have
aH4H-1
BI
Q44
Q441)
1
1/2
nM+#-1
1/2
Σ
Σ
Llllabe
a²
(=EH
+TH
G
B
1/2
-
1/2
Σ t=nlitty
a
We now the final term follows. For timesteps with small changes in the environment, Lemma and for timesteps with large changes in the environment, use naive upper bound. Define the set of timesteps To == {t ||r+1 relloc I 8n. || - Pr I 5p).
I6
a
[1g Q E
1+1
(c)
E [[g
Q
M++1
Te+11
+
403
a
a
t=n
TE
To
(d)
o 02
3G 33
4U3
LETo
TT
(e)
3G
3 3
4U Rm
4U (
a
a
a
adR
adp
W1A 2/3
1/3
W24
2/3
3G2 B2 P.H. (n+ D
1/3
(16)
a
where (c) follows from the Lemma 24
(d) follows from Lemma and (e) is obtained by choosing
ôn = AU3 AlßARna An,nn
1/3 and 8p ( (AUB Au3Apanncnnpp 1/3
and defining Wi (33 2/3
3G
3G7
W2 (AU3))(
We substitute the bounds I,..., Io (using Proposition 4 into 15 and use the squaring trick from Section C.3 in 33. The above equation is of the form, X Y + ZVx. Completing the squares and rearranging, we get X < 2Y + 7 Hence, we get the final result as
a
Σ
E [1
(=
4U3
2By3(TH +1)
2a(Ba + + +
2B R
+
a)
X
X
2B H1 A P.
8U2 7X2
4B, + +)2H
2D H
+
X2
4Ba(TH + +1)2 (m1HH
2D
8W3A 2/3
(n 1/3 722
+
X²
22
AW,42/3 2/3
P.H. (1+1)H 7X2
1/3
AU
4UgG
CBB333H
ad
ad
a²x²
2W, 2/3
11/3
22/3
2W, 2/3
H1/3
6G2 B332
P.tt.
(17)
αx
+ 3 (8H) + (((H) + 0 (AR + (A (p) + + + (?)
<
(n H1/3
2/3
2/3
(nnH 1}H 113
+0
+
+
72
7
26

---

Published in Transactions on Machine Learning Research (02/2026)
2/3 (nnH((++111
2/3
1/3
A (n
1/3
82 a²
+ Ô
+ +0
+ +0
α
a
where (.) hides constants and logarithmic terms.
D.3 Average Reward Estimation
In this section, we first analyze the gap between the average rewards and the rewards accumulate by NS-NAC in Proposition We then characterize the error in the average reward estimation in Proposition 4
Proposition 3. For any [N], if Assumplion 7 is satisfied, then the following holds true
nH+H-1
Σ
EJJ (s(. & D1B(TH + 1) -H)) + D2(TH + 1)2 (xH
where =1 + AUR AUn, D2 4UR Lp and (p Σ
(n+1)H Lt=nH
|P Prlloc.
Proof. Given time indicos conditioning on St- T Pi-r and rolling out by applying -e-r-1 Pt
O, recall the auxiliary Markov chain starting from 8t-7 constructed by
as
Tt-r-l Ot-r
Pe-ry 86-++1 mt-r-1 at
Pt-ra
8t Mt-r-1 āt
Pt-r,
Rt-r-1 ū++1.
81t1
Also, recall that the original Markov chain is
Tt
P. 11, ... St
m 1 az P4 9131 The a++1
St-r de-
St-++1
ac-++1
Further, recall - En,a - (o (a)))-a
We start by decompasing the term as
E [JT - rr(( [t)] E [ -
(18)
+ +E (re āt) re(st,a)) + EE Pi-eo - r3( az)
(19)
Note that Jn is the difference in the average rewards between the two policies n, - two different. environments rp) and (Pt-r, that share the same reward function. Hence, using Lemma and Lemma 3 successively, we get
1 S E
LT
Ti - Te-r-1122 Lp P. Pe-rlooo
< E
Lr
T; Ti-1 Lp Σ
- -P.-Pi-I 1
i=--
1=1-++
< L By + 1) Lp
(20)
where - E P; P.-P.-. P; Ioa-
For 1,, by Lemma 23 and Lemma successively, we get
< 2 2drv (P(()), F(Fr,,), P((r,, oz) -FFr-r))
< 4Un A Σ ||,, - T + + Σ
[P Pt-
i=t-T
27

---

Published in Transactions on Machine Learning Research (02/2026)
s 4Un/ + +)² AURT P,
(21)
Finally, bound I3 using Lemma 17
Is S 4Ummp'
(22)
Plugging the bounds 11, I2, Is into Equation
19 and setting ти min i 0/mp' 2 min
a))
nH+H-1
nH+H-1
Σ
B (e(s S
Σ
+) B2 +
LpApI-nH-Tnt3.
4T
(=nH++T
+ 1) P.HH(n+1) AURV 8| + (40 LT)
4URmp""
$ (L + RU 7 +
-
+ B - TH)-
Proposition 4. For anyy E [N], if Assumption 7 holds and 0 < < 1/2, then we have the following
nH+H-1
4
Σ
EJ - -m)
2BT +1) DLYTH +Bs((TH 1)²
(a
7
H=n
D13 H
4WA
2/3
RaH.(a+1)H H1/3
4 2/3 P.nH(an H1/
72
where D3 AURFs + 8U2, D1 B2, ARMH(n++) = (n+1)H
Ltent || relloe, PPNHH(N41) Elati [P++1 - Pelloo, W3 = (3U3) and W4 = (403))
Proof.
Recall that ot = - J Using the average reward update equation (line in Algorithm we have
2 = (nt + (rr, az) - 11) - J )
(0: + J 1 + (Tr at) 111))²
( (re(,,())) -T) I of + 2700 az) - 7;) 20h (J - 1 + 2(T - + - J + 20 (
(1 --27)0z + 27d
+2J - )2 + 27 (r: u1) - 11
(1 (1-21)) 1-27)00 + 27A Pt, Pt, 0.) 2. 1
+ +2(J, J + 2- (r:(e, ut) -1)-
= (11-2)) + + (,. P,, Fi, T/i Oz) +20 J 20 -
2 1 + (r:
32
Rearranging and setting = = minfi > 0/mp'-1 & min a)) we have
n
nH+H-1
E
oin1
n
Σ
Eio
Σ
Σ
E[A(T,, P P.. 0.)
2"
t=
12
nH - H-1
E
(J
JAL
nH+H- Σ
El
J J
Σ
7
t=
28

---

Published in Transactions on Machine Learning Research (02/2026)
nH - H-1
E [J 1
}
nH --1
Σ
(l(eee( 0x) nT.)9
18
We now analyze each of these terms starting with the first term as
B m H TH
o mH
20
1
<
27
By Lemma and the average reward update equation, we have
nH+#H11
12 <
Σ
Brß(TH + 1)² + Folme - - + PMTAPI-aM--atH
< ByB((T + HH -TH) 2U (H (T(TH 1)? 1)2 (n
By Lemma 6 we have
2
n
2U 2UnARnH((01111
2 RURAP,(4122)
T3 S
Σ
||+41 - re/lo + Lp|[Ptt1 -P/llo
< 7
+
?
1=
By Lemma e and Cauchy-Schie Schwar- inoquality, we have
nH --
1/2
nft + f1
[[ )2]
1/2
I S
Σ
Σ
72
+
Mtnn MTTH
-
L2 3
1/2
Σ
E(o2]
72
We now bound Is as follows. For timesteps with small changes in the environment, we use Lemma and for timesteps with large changes in the environment, we use a naive upper bound. Define the set of timesteps T = {t: [|r441 rellx= & 8R, PPt1
Pelloc s 6p}.
nH+H-1 Σ
E [J )]
15
t= TTH
E [
4U
Σ TęT,
IET1
(0)
382
313003
3T.2 B3
4U3
7
7
1ET3
1ETs
(6) 352 H
3L353H
3L2 B2
4U AUGAR
A (n+HH)
70R
70p
W R.HH(a+1)H
2/3
T/3
W
1/3 (m
312,3333
(23)
where (a) follows from Lemma G and (b) is obtained by choosing òn = (10
(WiAAnn 1/3
and 8p
37
U3A 4U2
1/3
and defining Wg (1U3))3 W₂ = (1UU))23
3
29

---

Published in Transactions on Machine Learning Research (02/2026)
For the final term, we have
Is S R - -TH).
Putting everything together, we have
a#+H-1
Σ
Eloī]
= +H
2
+ ByB(TH + 1H - -T) + 2UnFoYT H -TH) + B&(TH + +1)3 (n+
20 A (n PMH
nH+H-1
1/2
L2 B 1/2
Σ
Bio;]
72
MmnH+TH
W3A 2/3
Rn 1/3
2/3 W A
1/3
3
(m
+ +.00 TH).
7
Now, we use the squaring trick from Section C.3,
33 The above equation is of the form, X Y + ZVX.
Completing the squares and rearranging, we gel X < 2Y + Z Hence,
-
Σ
Eløi
4
+ +B, + () -T) + AUR Tn) + 2Ba + 1)2A (n+
40 ARARRM(D UUAARMH((HII)
92,
7
7
72
2/3 2W, 2/3
H1/3
2W, A 2/3
H1/3
(.
+ +UUR>(H - TH)
4U2
+ 2B, 1)2 TH) + FURFFGTHHHH TH) + +BglTT 1)2 A (.
9L B3
4W, A 2/3
R. H13
4W 2/3
P. 1)H H1/3
+ 8U R - -H).
7
D.4 Technical Lemmas
D.4.1
Actor
Lemma 2. If Assumption holds, for any t, 2 0, we have
Σ EiT log (s) V.T (.T((s)
J J
B C
<
C + 1 -Qi +
| relloo Lp|[Pr+1 Pelloo
where Zs(s) (xl1)) is defined in Assumption and other constants in Section C
Proof. We have
J J
30

---

Published in Transactions on Machine Learning Research (02/2026)
[+1
+
J
J Po
(0) 1841
f Σ
(S)Tee -V. +Q(ssa) Qc(s,a)
8,0
(6)
1+1
Jt
+
d (8) (s) (s) 10, Vi
log (1)))
1
log
t+ (a|s)
(als) - Q((s,a)
8
B
π
(6)
log Zz(s)
J
Jr
d d (ss)eet
Q7 (s, - -v (s)
(s(saa)
3
6,8
log Z((s) a((s,a)
2
Me-1 - Jett
Σ
Pt (s))T((aas))
B
8,0
+ Σ Pt
(s)( (alss)) [QT (8, a) (c(s.a)
(24)
8,0
where (a) follows from Lemma (b) follows from the actor update (line 10 in Algorithm D and (c) is due to the non-negativity of KI--Dveegence.
Next, we bound the last two terms in 24 Under Assumption
!
we have
d (s)
log Z((s)
1= Σ Pv()
(a(ss))
Q((8,a)
Pe
B
log (s(s)
P.. (s)s)
Q((),a)
B
8,0
log (s(s)
thh Pe P (s)
(s)
+ +C 1
Ed (s) (Q] (s. Q)(s
(25)
8,0
Further, we have by 1-Lipschivzness of the tabular softmax policy
L 2UQ Σ
P =<(ass)) 2 22 BUQ A 2 -B,3.
(26)
Plugging the bounds from
25 and
26
into
24 and rearranging, we have
Σ EmisPe(s)
log Ze (s)
V. (8)
3
14
Σ
d (als) Q7 (s,a))
+ (
Big + C
6,0
M(+1
J
B.3
|1 rrlloc
Lp Lp||P - -
1+1
|
Ool +
C
C
where the last inequality follows from Lemma 6
Lemma 3. For t 2 O, policy m. satisfies
| Tell2 & B23
where B₂ UQ-
31

---

Published in Transactions on Machine Learning Research (02/2026)
Proof. By 1-Lipschitzness of the softmax parameterization of the actor 54 and Lemma 24 have
|T++ Te
|| s BUg.
Lemma 4 (Average-Rewar Reward Performance Difference Lemma 41 The average rewards for any two policies n, n at lime salisfy
Jr J
dT..Pe(s
Σ
T(ass)
@F (s, V, (5()))
#ES
aEA
Lemma 5. For any 1, 2 0, holds that
J
||re le UQl|P, Pr
where ni represents the optimal policy for MDP A,P.s.,PP..r.))
Proof. Consider the linear programming formulation of an MDP M(S.A,PP. 55
min J (s(s))
such that J + V(8) 2 r() - Σ
P(s'ls, V.
(27)
If the optimal solution for Ms is Jú, V2, we have
J71 rr + 1VV
Now for A,P.,,,P we know
J < |r, + (P, 1 1)V
< Ir + (Pr I)V; + (r-- - ry) + (P, Pv) Ve
< | 1lo + + + PP))
Hence, we have
J J7 < |r: - Te loc + PP. V7
J. J < ||r:--ra loo + -
Lemma 6. There erist constants L 1)) and Lp nURM such that for all policies π. and timesteps t, L, it holds that
J J
J S L1
T' ||r, rvlloo Lp -relxx
Proof.
J J
J
J
+
J
J
(28)
T1 Ta
where Ti is the difference in the average rewards between two policies n, n' under the same environments (re, P.), while 72 is the difference in the average rewards with the same policy n', but under two different environments (re, P.) and (re, Pr).
T = J J
Eyngr Pr aemms Pr (re(s, rr(ss'a'))
32

---

Published in Transactions on Machine Learning Research (02/2026)
RdV (d @ d 8 8T
(a) T -tllTTTo - n 1|,
(29)
where (a) follows from Lemma 22
where & denotes the Kronecker product. Next, bound T2.
T2= J
J
(s)((a))))) - d= (s)T((()))r))))
8,8
d" P. =
d ( Pe Pt P
8,0
K rr re 100 + AURRTVV T Po OT' d' Pe 87)
|r. lr LP P
(30)
where (6) also follows from Lemma 22 result.
Substituting the bounds from
29 and
30 into
28
we get the
D.4.2 Critic
Lemma For any policies æ, n², we have
| Q ll2 S
SG.l π T'/12
where Gx = 1U4
Proof.
(s, (0)
rr(s, J +BUNP() (s'))
8Q7 (s.a)
-OJØ
(v,
P.(''ss a)
Ūn
s'ES
8QT (s.a) a)
2
(31)
Ôn
Ôπ
2
2
aQT (s.
(b)
<
2
d (s)0 S 2UQ
(32)
Ôπ
It follows from mean-value theorem that
1Q7 (s) QT' (s, a)l s 2Ue| π T'12. for all
QQ QT ll2 S G | - n 12:
where (a) is by using the Bellman equation, and (b) follows from Policy Gradient Theorem 20 and
Lemma 24
Lemma 8. For any timesteps t, 2 0, we have
[E [QT Q ||2 s GRIlE, Fr + GrllP, - Pello
where GR 2X- //S|AA| and Gp Lp AURA~ - AUR (M + 1))
Proof. Recall the diagonal matrix D..P. diag (dm (s) where d (.) denotes the stationary distribution induced over the states, while denotes the all ones vector. E denotes the subspace orthogonal to the all ones vector. Pseudo-inverse of matrix is represented by Xi. Now, we have
mE QT Q71 ((*A::))
mm (J rr.) (A=PP)t DT (J -rw)lla
33

---

Published in Transactions on Machine Learning Research (02/2026)
( re)
DT.. - D Dr (J re)ll
(A
(ATP.))
Prr - DTrPerrll
IIAAT.:P) (JØ1- (A A DT. (J71 rre)ll
9
3 X-' (IID (J 1 | D 1/ 12+ + | D Derelll)l))
|I(AT.P) y1 D (J (A DT. (J - rrr)l
A (ViSIIA) P: ||2]_ - + | Dr 2 U S||AA| + |D D
I(AT..)) y (J71 --) (A P. pr) DT. (J rr)
3 A SIIA (1]r:-r,)) rr + Lp L - Pr + RdRTV (d (d
@ π d 8 T))
A- 'IDT.P - -D).P:'r'
(33)
(A P.P.P... P. (JT1 Tvr) (AT.Pr) DT..)tm.PPPP.. (A
A- VISA (||r, - 10 + Lp Pr Pr/ox + 2UR MPPPP P. - Prlloo)
A DID... Tt- DT.P''rr
(34)
P(AT..))) yt PT (J - r.)) - (A A. P. (J -
3 - VISIIA (2 r| - 11 + Pp - Prll + 2 P, - Pr
|[(TTP:)) P. (JØ1 rw) (A T. DT. (J71 -re)ll2
rw)
A VIS||A| (2)
- Tp 1100 + Lp Prll + 4U PP - Pr)l
|| P.). P.JT PA 2
2 X- -- /VISS/A |A| (2 - r 100 + Lp pP P.
Pr + P:-rll))
2U P - P..p.1l2
(
$ - V (2 loc LP P. Pelloc + AURM|| Prllo))
2 2(M + +1) P. - -P
&GRlIT, Telooo + Pp Pr
where (a) is because E [r( J(O) + A(O)) 0 (see TD limiting point (3 in Section 5.1 (b) is from
Lemma. 0 (c) is by Temma 6 (d) is due to Lemma 22
(e) is using the same process as the last. step for
the second term, (f) is because | Y < x Y)Y S | Y| Y and (g) is by
Lemma 24 and Lemma 22
Lemma 9. For any t 2 0, we have
g Q QT
lla S GRllie+1 - rrlloc + Gpll1P41 - Prlloc + ByB.
See Section C for conslants.
Proof.
| [Q
Q. 122 < | Q Q. ||2 1 [Q..t Q, l|2
Gnl(T - ry + Prt Pelloc + Gz - TT/l2
Gallrest - rxllo + Gr - -Pelloc + B23
where (a) is by Lemma 8 and Lemma and (b) is from Lemma 3
34

---

Published in Transactions on Machine Learning Research (02/2026)
Lemma 10. If Assumption 1 holds, for any T, we have
(T P.. To. 4r 0.) - BT 1)2 BqaT + BB Bo
where B3 (Fin + F2G + || B4 (2 B, P2GR and Bs Pip+PaGp-P3
11lo and P Ei P - Σ Ei rr
Proof. Recall from Section the definition
[( P,r, 0) T (r(O) J. A(0) + T (A(O) AT.P) sb.
We first decompose r(.) into the following four terms
(T,(T, Pt, th,, 0.)] < E [T(T,, Pt, r,, 1r 0.) - ( Pt-r, rt, 1i. 0:)
E [ P. Pi-.Ire 2r. 01) [(T Pt-r, Po P $1-T:
12
E [ Pon,
In, V1, 0₁) -ITT--r-,, O.)
13
E [[ t-r Fr, Vi Õ₁]
We now bound each term as follows.
(a)
I S Fix
T| - Te---1llz]
Fip P. Pe-rllo
< Fi Σ ||, - -122 + Fip
Σ i=(-++
P
(M)
< Fix B23(T + 1) FipAp1-741.t
where (a) by Lemma 13 and (b) is duc Lemma
12
For the second term, we have
(c)
1 S
E2 [lvb. - 1hrr1112 F2E
Σ
1 - 3
i=1--+1
(d)
F2
(2UR 2U2)a GRllI: ri- + Gpl Pi- B23
li=t-++1
< F2(2UR + F F2GrAp,
F2GrBB3r F.G
where (c) is by Lemma (d) follows from Lemma 24
QQ+11 - Q1112 S B(UR + UQ) by the critic update
equation Lemma and Lemma 12
We also define A ARI-T-1.- = Ei ||;- r lloe and Ap =
Ei P. P
For the third term, have
Ig & Fav
Σ nT: n -----1l1
2/12 T
Fi_,
+ F3 Σ i=t-T
- -
S & A 1)2 PaTApp--4i.
where (e) is duc to Lemma 15 and (f) follows from Lemma
12
For the last term, by Lemma 16
we have
L Famp".
We get the final result. by putting all the four terms together.
35

---

Published in Transactions on Machine Learning Research (02/2026)
D.4.3 Average Reward Estimation
Lemma 11. If Assumption
1 holds, for any > T, we have
[| T,, ne, 0+) - (T + 1)² + Felm: -h-Tl++ Bgg A P
where By (FoL* + F/v 8 B2) B3 FiLp and Apo-r+i, D
Σ Eist-r41 P: Pi-
Proof. Recall from Section B the definition
P,,,,, -n
We first decompose A rt, T/., Ot) into the following four terms
E|A(TT Pe 0,) BlA(T ne P., O.) A(TT---1:: 1..----..P--- Po-,, 0:)
+ P me Oi) A(T Fe, Ph
F1, E ([A(Ti---111 r4, 0,) 1h O.)
P To M-TI Õ,)
We now bound each term as follows.
(a)
1 x FIL E [|| T + FgLp P. - Pi-rllo
Fg L E
|| TI- 1|
FFLp
P P
(b)
< F3 B2 + 1) FsIpArs-rt1.
where (a) follows from Lemma 18 and (b) is due to Lemma
12
For the second term 12, we have
(0) S Felm: m--l
I2
where (c) is by Lemma 19 For the third term 13, we have
(d) Is I Fry FIVISIA Σ | n T1- 112 Fi
+ Fr Σ PP P
(c)
P.S/SAABB 1)2 F.AP
where (d) is due to Lemma 20 and (e) follows from Lemma
12
For the last term, by Lemma 21
we have
1 < Esmp".
We get the final result by putting all the four terms together.
D.5
Auxiliary Lemmas
D.5.1 Actor
Lemma 12. For any timesteps > >>>0 O, the policies generated by Algorithm ] satisfy
Σ | n πt--- S B2 + 1)2
36

---

Published in Transactions on Machine Learning Research (02/2026)
and reward and transition probability matrices satisfy
[r r 1100 ST
Σ
|| r-110x
i=t-r
imt-741
Σ P ST Σ
PP-PPi P
i=--++
Proof. By triangle inequality, we have
T-- - Σ E TΙ -T
n3 T
i=t-7 j=t-t
|r -
i= j=t-T
(0)
S B2 1)
where (a) is by Lemma
The rest follow similarly using triangle inequality.
D.5.2 Critic
Lemma 13. For any n, n P, P', r, tb and O = (s, a, s',a'), we have
[( P.r, Φ. 0) - I( rb O) & Fi | - n' + Fip P
where Fix = 2U2 W +
Fip = + +1 ())
Proof.
[(TT, P.r, 1b 0) T( P', 2/ 0)
= 22b (J .'' (0) (('))
J P.r Prr (0)) + + T A(O) - + +2b 2 (ATIP' A T. 2b}
|| TT pr - JI + | |A())) QT. _ Q . QT'PPrr\
+ 11 00 AT'PP' A
||bbl12
(
Qr'PP'r' P + 12b ||(0)) ( -
2Uq T || - T'|l2 + 2U Lp
2
+ +1
A T.P
||
(2)
< 2U bZ π - T'Il2 + 2U Lp P P + 4Ug Gz - T' + 4UGGP||P P'
+ 11 A A P.
|
2UQ L
T'/12 2 2U
PP PP
4U 4UQG - 4UpGr |P -P'lo
+ +20g 2dTV (
P'
JT.
P
(0)
& 200L | - 22212 + 20 P P'
W - 1|2 + GU P
+ 8U + 2 - 2 + 8U 1) 8 A π
1100
where (a) follows from Holder's inequality; (b) due to Lemma
(c) by Lemma and Lemma 24
(O)A 1); (d) is by Lemma 24 and (e) uses Lemma
22
37

---

Published in Transactions on Machine Learning Research (02/2026)
Lemma 14. For any n, P.r, w. v and O (s, a, s', d'), we have
,,,,, 0) [(T,, P,r, 0 Fallab 1/'1/2
where F22 2UR 18UQ.
Proof.
[( v, -(T)) +'O0)
- ([](())l2 + [J)PP(())l + [A( QT. 11 l
+ +[(AO)) - -A lll (11vbl2 + (1
$ (2Un + 18U2)l 12
Lemma 15. Consider observation from the original Markow chain by O (st, (4 81+1, a(+1) and auciliory
Markov chain by Ö (st, de, 8141 a.+1). Conditioned Fi (s--T, T Pt-r]: we have
E [T Pe-r. re, - Os) -(TT----1 Pe- Ft, 2,,,, 01)
|| 112 F3 Σ PP-rlle Σ
i=!-T
where F3 16URU///// 24U VSSAA
Proof. Consider the original and auxiliary Markov chains whose construction is described Section
[(T Ph ( Pt- ti. F.
+1, (O.) r., J (O:) - L.P. Fi-r]
+ + E [A(OO)) A(O,)) Q IP, ri [
+ WT E [(((O.) - A(Ō))) Fi-r] th-
< 11b111100
||E [r:(O.) ((,) + JJ, 1 (0) -0
+ +1
| [A(O.) A(0.) -1
+ +1 [A(A)) - - = 11
< 2Ug 4Un 2drv (P(O, -1F--)) P(O: E ((F-t))
+ +2Uq 4drv (P(Or E ( (F--), (F--r)) UoV IA|A
+ +2Uq 4drv (P(O) E FF..))), P(O, E (F(F-))) ( |S||A|
< (16UnUQ + 24U3 24 VISIAA)
\VIS|LAB
|| ----- Fi + Σ Σ E=
[P; P Po-rllo))
==
where the last inequality is from Lemina
23
Lemma 16. Consider an observation from the original Markov chain by Or (st, at, 8141, a(+1) and auriliory Markov chain by O, (Bt, ãt, 8141, āz+1). Condilioned on Fi-r {se-T, Tt r- Pt-r), we have
[( P.-r the Ôz) S Famp"
where F1 = 8URU + 24U3 / S|[A|.
Proof. Consider the original and auxiliary Markov chains whose construction is described in Section Also, consider the observation tuple O (8/: a/ 8441) where s dmI-r-i.P:-r(1 of (
38

---

Published in Transactions on Machine Learning Research (02/2026)
$/11 Pr-r(-Isk, ab) and 4/41 me-r-1(-16(41) From the definition of I(+) and the TD limit point equation 3 it follows that
E Po F
Hence, we have
E
Pr. T Te ( Fr, 2 Oi
E [ P. Pi-r, F,, 1-.
$ 1|
EE [rr(O.) - - (0) (0,)
11
| [(A(()) - A(O)) Q QT Q. [r-->]l1 F,-
1| oo || [(A(Ō)) - A(O)) 11-
S 20Q- AUR 2drv (P(Ō, E (F)-)),, E (.FF-r))
2UQ 4drV (P(O,
E (Fi-)) LiQ UUQV/SIIAA
+ +UQQ Adrv (P(Ô, E ((Frr), P(O, E ((F 2UQ VSIAA
F4 Σ
IP ( (s))))a-----
8,0,5',0'
=
F4
Σ
Tt (0=
P(3 s(Fr-)) P(s! s s(Fi-+)
8,0,8/,ol
Σ P( F(F( -
S
Famp
where the last inequality follows from Assumption
D.5.3 Average Reward Estimation
Lemma 17. Consider an observation from the original Markov chain by O2 (st, a2 8/, at) and auriliary Markov chain by O (5t, âz, $141, ā.+1). Conditioned Fi-r {st-r, - P p have
E [T
a.) az) s RUp
where J = E d (
Proof. Consider the observation tuple Oj = (5/, di, S d(41) where si d (.), of (--r-1(-1()), Then, by definition of PP we have
1/41 P.-r(1ls,, at) and d/41 n (-16%
E M m - re(s), -]
Hence, we have
PL az) F(-r]
E -
- re (1 de) - (8) + a()
E (ri(ssh, df) - ri (8t, (z)
< 2 2drv (dm---- @ TT-T P((r, ā.) E -(F
nUndrV (d----- P(št E (Fr-)))
(6)
2 mmp
where (a) follows from Lemma. B.1 in and (b) is by Assumption
1
39

---

Published in Transactions on Machine Learning Research (02/2026)
Lemma 18. For any n, T', P. P.P', ,r,, and O (s, a, d'), we have
- - ,,,, 0)1 |||55000 FoT.T| x
x'1l2 l12 + Fo po
where F5 nUR-
Proof.
MA Prr, A( - (r,,,,
< (17 Jr. ,rr.Prr - -(1 J J ( JTT P'r)
$ (1 - JT. (r...r)(((ss4))) - jr.P.r) - -11 J.P.P. jr(8,) - jJm '''r))
+ 1(1 -T)...)(((ss))) a) -Jx' p' (17 JTT (r a) J''''r
< <4UR|JT.Pr. J p (a)
AURLm π x 1 - AURLp||P P'
where (a) follows from Lemma 6
Lemma 19. For any n, P, r, th of and = (s, a, 8, aa), we have
[M P,,,P,r,n,O0 0) - A AT, pr n/ 0)1 S Feln - n'l
where F6 R-
Proof. Recall the definition of A(-) in Section
B
It is straightforward to that
- A(T,
- v/'l
Lemma 20. Consider an observation from the original Markov chain by O2 (st, U2 84+1, ac+1) and auciliory
Markov chain by Ö = (5t, āe, $141, at+1). Conditioned F. (S--1, -1- P.-r), we have
P E [A Pt Pe-ri Tts BT- Ot) 11 Fe Fr
< AS
|| - F Fr Σ Σ List-r
[P. P Po-
i=t-r
where F7 SU?-
Proof.
E A( P.-,, m.: 0₂) - Po 0h F.)
(1 J
IP r:t1 :: )B [rr (56, 0z) F(-r]
2UR AURdTV (P(O, E [F P(O; E =(F(-r))
(a) FAVISILABE
TT n 1 Fr Σ 1-t-
+ F7 Σ
PP - Pi-rlx
i=t-r
where (a) follows from Lemma
Lemma 21. Consider an observation from the original Markow chain by Oz = (82, ats 81+1, a(+1) and auxiliory Markov chain by Ö₂ = (5t, 0e, S141, ar+1). Conditioned on Fe {St-T, Te --1 Pt-rh, we have
E [A P Pt, Tr - 0:) S FSMP
where Fg 8U2
40

---

Published in Transactions on Machine Learning Research (02/2026)
Proof. Consider the observation tuple 0j = (5 86-1 where oy d dj ~ (T-(-1((15(), 8/41 Pr-r(11o), ai) and 4/41 11164)(1))411))
We know
E M. [ =0.
Hence, we have
[A(T ( Pe- T, [r-
= E [A(Te-T-1y Pt, 11- 0:) - A(F+ tt 10
E [ J 1 P (T)((, a.) -
(()
2 AURdTV (d @ -1, (( āe) E -1Fi-r))
(0)
$ 2 AURTTV (d P(e E (Fr-r)
(6)
2 8U3mpp
where (a) follows from Lemma B.1 in 33 and (b) is by Assumption 1
D.6 Preliminary Lemmas
Lemma 22. For any policies n, n, nt and transition probabilities matrices P.P', it holds that
dTv drv (TT.P. (Tm d .PP
- M &MVISIAI π 12 + MIP P'lloc:
drv ( 8 n¹)
< (M + 11 T 112 M - -P'
drv (T OTOPP n & P
< (M + 1) ||||||||| T- (M + 1) P 10.0
drv T P T
s (M + 2) [ (M 1) P'
where @ denoles the Kronecker producl, and M = (nog, m-1 + 1
Proof. Recall that P (.) is the stationary distribution induced over the states by a Markov chain with transition probabilities P following policy л. Define the matrices K, K' E RíS)x(š| such that K(s,8') = Eoes P(s'ls, (a)s) and ('(s, s')
P' [[[s'|s,a)rti (a) Further denote the total variation norm 1 Note that [P P' max Σ, [(s a)))ss
8,0
From Theorem 3.1 in 56 we have,
// ((ds)(K-KK>))))) K'((s)) TV
drv (TT. d
< M sup 1llrv=1
1ellz== 1 1/8 ((ds)( K') ds')
M sup
(Kds)(K-KK)6
/ [d(ss) Σ
< M sup
Igllv==1
aaA
M sup
1gllr==1 JS |
/ Σ P((ds) P' (s))s,)))()l (c|s)
+ sup
41

---

Published in Transactions on Machine Learning Research (02/2026)
s MM/IS/AA|I 22|22 MP P'llo
For the second inequality, we have,
dTv (dt TT, d p (
Σ d (ds) P (ds))
2 J
$ 5
Σ
T (ds)T(als)) dT. P (ds)
d (ds) - dth PP' (ds) (a|s) (als))
+
π 112 + dTv dd dt aTt
< (M + / π n'' M - P'lloo
The rest follow in similar manner.
Lemma 23. Consider observations O2 (st. a1, St+1, a(+1) and Ô, (5 Ôe, 1++110++11 and define Fi-r
{8--T, P.r). We have
drv (P0000006//-/-)1000::.-1-. VSIAA Σ B|I - + [P, Pi-rlloo
Proof.
drv (P(O: E (Fi-r), P(O, E P(O, ((i--))
Hi
Σ
P( 8,0, = 8141 = s',att1 = d' F(( - P(3; = =s =a, 3,41 = s', s',ā++1 = a' F(-))
2 s.a,s',a'
P((: 8,0, F4
Hel
2
8,0,8', 'a'
P(3 = s, = Pr (1
1
Σ
P(s 8, =3 Hel
2
4,0,8',0
P(; = 3 = aF 1777
Σ
= 8, PPis==a
8,0,8',a'
P( 0 = 121
Σ
8, a)
==-r=2(d'(s))
2 a'
P(s 8,0, = - P(3, = á = a(Fe-r)
8,0
Σ P(., 8 = afF ) (dd
(5
5,0,''a' a''
$ V || Tt Te-- 212 For + drv (F-)), (FFrr) + [P, - P
Finally, recursing backwards until yields the result.
Lemma 24. IJ observation is denoted O as, ,d),, then the following hold for all t, !
42

---

Published in Transactions on Machine Learning Research (02/2026)
1. ||T UQi |QQll2 Ro UQQ
||([O]| 2; |A(O)] S V2
||A.P A co00 $ 2dTV (d..P. &
n d"' ''' B P )')
|| 11l12 Qu4+ Q:112 + +Q 2.122
Proof. We have the following.
1. See the projection operator ng (+) used in Algorithm and discussed further in Section 5.1
2. Follows from the definition of AO)) in Section 5.1
Follows from the definition of A T.P in Section and
A T. A '''' 1100 max
(s.0))aaaaapP((/ssssmmm 8'
d
P
(8, (dls)
4. By the definition of sb. 1 (Q: QT7 and triangle inequality
D.7 Universal Dynamic Regret
While dynamic regret defined in Equation remains the predominant metric of performance in non- stationary RL literature [2
we additionally consider the universal dynamic regret often used for completeness. We now present upper bound the universal dynamic regret defined as the difference between the optimal total reward that can be obtained in T time-steps and the reward accumulated by our algorithm as follows
47 IN
17
in the adversarial learning literature 57
58 59
T-1 Σ
T Σ
U-Dyn-Reg(M,IT)-
max Eng (#))
re(sE at)
re(S,, ai)
(35)
Observe that this notion of regret is slightly different from the dynamic regret defined previously in Equation which compares against the sum of average rewards J. obtained by the policies πं.
We make the following additional assumptions on the structure of the MDP. The condition number captures the sensitivity of the stationary distribution to the change in probabilities and the recurrent coefficient. captures how often state is visited. Assumption (Bounded Condition Number, induced by the transition probability matrix pT [|S] is defined the maximum norm of the Drazin inverse of 1 p as
The condition number of Markov chain following
<(P") 11(T - P max
where ## represents the Drazin inverse of matrix B and |B|mmax maxi.j Brj- We assume that for all environments E [T], there exists k > 0, such that the condition numbers of Markov chain M, following the optimal policy of are bounded as
<(PT)) S
43

---

Published in Transactions on Machine Learning Research (02/2026)
Assumption 3 (Bounded Recurrent Coefficient, 61 Denote the first passage time of state from state in an MDP M(S, P. following policy и as Yijj The recurrent coefficient of state j, denoted by My. is defined as the probability that the first passage time YTJ of state from any state following any policy is smaller than the number of states as
(M(M)
min Prob (Y $ ||S)).
(,
We assume that for all environments t e [T] and all states j S, there exists v, such that the recurrent coefficients (M)) are bounded
M)(M 2 v.
Corollary 1. If Assumption 1 choice of porameters, then
Assumption
Assumption 3 are satisfied Algorithm 1 with appropriate
U-Dyn-Regh Remn-RegMMM.33 M, max E (m)) P-1
Σ re T()
E
Σ re ar)
s
Proof. We decompose the regret as
T
U-Dyn-Reg(MM
max
re (5 ((fmmm) m
-E
(1)
7-1
T-1
T
T
re an) ))
max
E
E
J
+
E
E
Σ
-1 (m:))
(a)
AT)
+
E
J
E
Te( oz)
1=0
(b)
Ar) + +
(15)\/2/A11/
A 1/673/0
where (a) follows from Lemma (b), Corollary 2, Theorem 1 in 50 and (b) follows from Theorem
E NS-NAC with Function Approximation
In this section, we present the NS-NAC algorithm with function approximated policy and the state-action value function and the associated regret bound. Consider the policy He parameterized by O Rd. Consider the state-action value function Q QTa (s, approximated
linear function f (s, a)w where fo (s, a) denotes the feature vector and Rdd We assume the actor and the critic function approxinations to be compatible as fo(s, a) = Vo log (als)
62 63
The natural policy gradient [00 can hence be expressed as
0:1 - 0, + 3Fg. Es,a [fo, (s, a) (s, a)w,)]
where wa. arg min [(Q
(s, a)
f0, (s, ]. In the absence of the information of the exact gradient,
the actor update step corresponding line 10 thus becomes
0 0, B.r.
The TD update step of the critic in line 9 can be written as
W++1 + we a [ra(se, 0z) - 17 + ff (St+1, 0441)44 - fe (st, a ar)w)] fe(se, az).
We now detail the assumptions under which the following upper bound on the dynamic regret of NS-NAC with function approximation holds.
44

---

Published in Transactions on Machine Learning Research (02/2026)
Assumption 4 (Uniform Ergodicity). A Markov chain generated by implementing policy me and transition probabilities P is called uniformly ergodic, if there exists >> 0 and p E (0, 1) such that
drv (P(s, E -1so = s), d S mp Vr 2 0, E
where d is the stationary distribution induced over the states. We assume Markov chains induced by all potential policies Te, in all environments Pz, & [7], are uniformly ergodic.
Assumption 5. For all potential parameters 0, in all environments Pe, [T], the maximum cigenvalue of matrix AT = E,,o,s','' (((),a)(S((<',e))))) a') fi(8, a] is -X
Assumption 6 (Smoothness and Boundedness). For any 0' e Rd and any state-action pair s E S, a E A, there exist positive constants LA, Le such that
1. || la 1,
al) (s, Lo110 0111,, and
||T0(-18) - (+1s))l < L 111
Definition Define the compatible linear function approximation error as
Eapp == max min Es Pt aame (1, (8,
0
The dynamic regret achieved by NS-NAC with function approximation described above can be upper bounded follows.
Proposition 5. If assumptions and are satisfied, capp the function approzimation error defined in and the parameters of NS-NAC with d d-dimensioned function approzimation are chosen optimally, then
T
Σ J re(s), ac) = =0 + -0(1/2 (ap t=0
Dyn,Re
E
Proof. Function approximation has been used commonly in actor-critic 34 33 and natural actor-critic algorithms in the infinile-horizon average reward setting. For the sake of brevity, choose not to repeat the proof here and instead we point the readers to 43 for the technique to incorporate function approximation into our analysis of NS-NAC detailed in Scction Section D.6 above. Note that the structure of the proof including the methods of actor, critic and average reward analyses remains the same with the only difference lying in accounting for the function approximation error capp-
F Regret Analysis: BORL-NS-NAC
Theorem 5. If Assumplion is salisfied, the lime horizon T is divided into epochs of length W = O(T2/3) in Algorithm then we have for any jt E 10,1,".. ,[,"]]
Den-Reg(M,TT) 8(
(
IT W
+
TNT W
+
T
NtW at
BtT at
+ +
TVB
(36)
W
of hedging EXP33P
effeet of initialization
cumullative change in policy over horison T
+
8TT 7
o T (
+
N!W W
(Tval)
+
ArW NI
+
A 1//3///3
+
1//3///3
Val
V7
cumulatioe change critic estimates
error in average reward estimate at critic
error due to non-stationarity
45

---

Published in Transactions on Machine Learning Research (02/2026)
where ÅT = A al -1 =0 / 1/3
, = (To' (Tolstur) 1/2 , = W (ri
5/6
and (.)
T
hides constants and logarithmic dependence on time horizon T. Choosing optimal value of j and resulting optimal porameters as a² = --- = ( 1/3 3 = (Ar)
W = - T/0T1/6 we upper bound the regret as Dyn-Reg(MM (M, S (11 5 |
1/2 and N
1 76/6
Proof. We start by decomposing the regret, for any j E {0,1,-1 ... [nn]] and corresponding step-sizes at, 7t,, St and number of restarts N as follows
[T/W]
(1+1)W-1
(M = Σ
E
Σ
Sei - re(s),
1=
t=
[T/
(111W-1
T/W
Σ
J
R
+ Σ
E B
Re
=
1=WW
T=
(4)
[T/W]
(1+1)W-1
Σ
J
Rest
+ +
WWin'
W
InT.
T
W
!=W
[T/W]
NIW
3' at
W (WVtt) + O (8IW)
- + O (3ttt)
1
+
3t
at
at
NtW ?
+ Ô
(WVat)
+ AWN,
)
+
A 1/3
(iw W2/3
N
Vat
AW.+-WW W2/3
1/3
+ W WWVnT
IT
W
TNI
+
N
+ +
B'T
+ +0 (TV
+ Ô
B
+ +0 (OTTT7)
W
W
at
at
7
T
NEW
+ + ( ArW
+ ///3
+ (TVat
W
Nt
Vai
1/3//2/3
T W
+
WW
In
where (a) follows from the EXP3.P regret bound of an
bandit with rewards in [0, W UR] detailed in Section 3.2 of 50, (b) follows from Theorem and (c) follows from Jensen's 's incquality.
Further, there exists some jt such that
[1
1/2
AT
1/2
T TU++Y)
1/2
<
T
T
T
1/3
[A
1/3
(T(1+11/(aaT)
1/3
S
=
S
T
T
T
(TTt/DNTT) 5/6
L
N*
5/6 < (TO*+)//OTT)
3/6
T-/6
W
We conclude the proof by adapting 3* a² , No to 3t, at, that [I][ O11) results in the final upper bound presented in the theorem.
at,n', nt,,ttt Nt in the above regret expression and observing
46

---

Published in Transactions on Machine Learning Research (02/2026)
G Simulations
Synthetic Environment. We empirically evaluate the performance of our algorithms on a synthetic non-stationary MDP, comparing it with three baseline algorithms: SW-UCRL2-CW bT1 Var-UCRL2 [3], and RestartQ-UCB (16).. The synthetic MDP environment simulates non-stationary dynamics by alternating between two sets of transition matrices and reward functions over the time horizon T.. The switching frequency, controlled by Rewitches: determines the degree of non-stationarity and the variation budget Árt for transitions and AR. for rewards. The MDP consists of [S] states and [A] actions per state, with two sets of transition probabilities and rewards sampled at initialization. Further, to benchmark the effect of the dynamic changes, the optimal policy is recalculated at each switching step Eewitch by solving a linear programming problem 15
The environment alternates belween these two sets of transitions and rewards, (Pi, and (P2, every T/ s/tswithhes steps. The transition probabilities, P1 and P2, are drawn from Dirichlet distribution with a concentration parameter set to 0.5, ensuring a moderate degree of randomness in the state transitions. The first reward matrix is drawn from Beta distribution with shape parameters to rewards spread across the interval [0, I}, with a higher probability near the extremes of 0 and 1. The second reward matrix r2 is sampled from a Beta distribution with shape parameters = 0.2 and 3 = 0.9, producing rewards skewed toward lower values, introducing diversity in the reward structure. We use random seeds to initialize the matrices, with standard deviation capturing variability across these runs.
0.5 and / 0.5, leading
Varying T. We evaluate the performance of different algorithms in the synthetic environment with |S| 50 and |A| 4 under varying time horizons T. Specifically, the time horizon T varied over the values 50 x 10², 70 x 10², 1000 x 103, 150 10*, 180 10 200 10², and 250 x 10 For each T, we set Rlwwttches 1000, resulting in a transition variation budget ArT = 303, indicating significant environmental changes across the time horizon. The reward function is kept stationary (no switching between rı and r2), and therefore An
Varying 47. We investigate the impact of changing variation budget by adjusting the number of switches Rewitches while keeping the number of states |S| 50, actions |A] = and the time horizon T 50 10 constant. The number of switches is varied across 10, 45, 100, and 1000, with both the reward function and the transition dynamics being non-stationary. The observed variation in rewards ART is 9, 48, 98, and 1000, respectively, and the observed variation in transitions APT is 4, 14, 30, and 303, respectively, corresponding to different levels non-stationarity.
16
16
NSNAC SW-DCRLD-CW
24
2/x
SW-UCRLECW Va--DRLL
NE-ACC
Dynanie Regret
2000-
0.1
0.06
Denanic Regret
NesUCRE2 RecaniQ-DC
Denamic Rogret
2xx 107
RestunQ-UCCB
10
1009
122 10
2,4 10
500-
11
1,2x
150
175
20
Time Step
XIP
[5]
AA
(a) [S] 50, [A| 50, Ar
(a) [A] 4, 1' 5 10
A7
14
(b) [S| 50, T 103, Ar 14
Figure 3: Performance of NS- NAC with different step-sizes in environment with 17 abrupt, randomly scheduled switches over T 4 x 10 steps.
Figure 2: Log-log plots showing the effect of varying: (a) number of
states [8], and (b) number of actions [A].
Varying [S]. We study the effect of varying the number of states while keeping the time horizon T, number of actions, and variation budget AT constant. Specifically, the time horizon 7 is fixed at 50 10 steps, and
47

---

Published in Transactions on Machine Learning Research (02/2026)
the number of states is varied across the values 100, 150, 175, and 200, corresponding to environments with different state sizes while keeping the number of actions fixed at 4. The Nawitches is adjusted to 75, 100, 120, and 150, respectively, in order to maintain a consistent PP, of around 14 for all environments. The reward function kept stationary with ART = 0 (no switching between r1 and r2).
Varying [A|. We examine the effect of varying the number of actions while keeping the time horizon T, number of states, and variation budget ÅT constant. Specifically, the time horizon T is fixed 50 10 steps, and the mumber of actions is varied across the values 5, 10, 20, and 25, corresponding to environments with different action sizes while keeping the number of states fixed at 50. The newwtthes is kept constant at 45 across all experiments to maintain a consistent variation budget PPT of around 14 for all environments. The reward function is kept stationary with A R.T = (no switching between ri and r2).
Parameters. The true variation budgets, ΛP, and ART, are provided to each algorithm, while the remaining hyperparameters are configured according to the optimal expressions derived in their respective papers. For SW-UCRL2-CW; the parameters include the window size W. and the confidence widening parameter Th both set using the optimal expressions given in the paper, and the confidence parameter 8 = 0.05, For Var-UCRL2, the true values of the variation budgets for transitions probabilities ΔPT and rewards ARr, along with the confidence paramcter 5 = 0.05, are used. In RestartQ-UCCB the ending times of the stages I,, confidence parameter are configured as described in the original paper with = (to adapt from episodic setting for which the algorithm is designed infinite horizon setting in our work). For NS-NAC, we tune the step-sizes and number of restarts by grid search. The effect of different choices of step-sizes can be observed in Figure Further, for BORI-NS-NAC, we set the number of epochs W [T2/3]_
0.05, initial number of samples No, and number of epochs D
NS-NAC
NS-NAC
3000
3000
SW-UCRL2-CW
SW-UCRL2-CW
Dynamic Regret
2500
VarUCRL2 RestartQ-UCB
Dynamic Regret
2500
Va-UCRL2 Restatt-UCCK
2000
1500
1000
1000
500
500
0
0
0.0
0.4
0.8
0.0
0.4
0.8
Time Step
x10
Time Step
x10*
(a) [6] 50, [A| 4, Ar 15
(b) 50,
|A| 4, Ar 0.06
Figure 4: Performance of NS-NAC and baseline algorithms in various non-stationary settings. (a) Dynamic regret for a single instance over T' 1 x 10t steps in an environment with 50 abrupt, randomly scheduled switches. (b) Dynamic regret for single instance over T 1 10 steps in environment with small, continuous changes.
Additional Environments.
We conducted further experiments to evaluate the adaptability of NS-NAC and baseline algorithms across diverse non-stationary settings. Figure ((a) illustrates performance in an environment with 50 abrupt and randomly scheduled switches (between P1 and P2), simulating scenarios with non-periodic unpredictability. Figure 4(b) captures performance in a continuously changing environment, where the transition from P1 to P: occurred gradually over T 10s steps resulting Ar 0.06. This scenario reflects real-world conditions where systems experience smooth drift rather than abrupt changes. The results highlight NS-NAC's effectiveness in handling both abrupt and gradual changes, consistently matching the performance of baseline methods.
48


Reviews:
[{"review_id": "pT8Uhar5WB", "reviewer": "Reviewer_yZXp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies non-stationary reinforcement learning in the infinite-horizon average-reward setting. In the considered non-stationary RL setting, the reward and transition functions may vary at any time step but subject to a $T$-step variation budget. Notably, the total variation budget, denoted by $\\Delta_T$, is of primary concern. The performance is measured via the notion of dynamic regret, which is relevant for the considered non-stationary setting. Such a regret definition is indeed a form of variational regret. The main contribution of the paper is an algorithm called NS-NAC, abbreviating Non-Stationary Natural Actor-Critic, which is a two-time scale natural policy-gradient method. NS-NAC achieves a dynamic regret scaling (in expectation) as $|\\mathcal S|^{1/2}|\\mathcal A|^{1/2} \\Delta_T^{1/6} T^{5/6}$, ignoring log-terms. In fact, the said regret bound for NS-NAC could be achieved only if the algorithm would know $\\Delta_T$ (or an upper bound thereof) to set its step-sizes and the number of restarts. Besides, NS-NAC relies on the knowledge of a projection radius $R_Q$.  \nTo relax the requirement of knowing $\\Delta_T$, the authors introduce BORL-NS-NAC, an *almost* parameter-free variant of NS-NAC that employs a bandit-over-RL approach ---BORL-NS-NAC still needs to know the projection radius $R_Q$. BORL-NS-NAC is shown to achieve the same (in order) regret bound as NS-NAC.\n\n\n**Strengths**\n\n- The paper studies an important and challenging problem in the domain of regret minimization in MDPs, which is both theoretically and practically relevant. The two key components of the model (average-reward criterion and non-stationary subject to a variation budget) impose much technical challenge. In contrast to existing work that are mostly model-based, the work presents the first policy-based methods, which makes a nice addition to the literature. \n\n- The regret analysis of NS-NAC appears to borrow its key technical elements from the analyses PG-style methods for (stationary) average-reward RL. Nevertheless, the analysis has its own complexity due to restart-based exploration and non-trivial learning rates as adapting factors. All these require requires some novel adaptations and tailoring in the analysis.  \n\n- Another key strength of the paper is to introduce and analyze its bandit-over-RL variant (BORL-NS-NAC), which removes the need to know $\\Delta_T$ or a related upper bound. However, the algorithm is not fully parameter free due its dependence on $R_Q$. Yet another strength (in my view) is reporting an empirical evaluation and comparison to existing algorithms.\n\n- The paper is written very well, admits a clear organization and presentation, and is overall a nice read. It offers a rich literature review, carefully positioning its contributions to the relevant work and citing most relevant papers I am aware of. Besides a few sentences that need some revision/polishing, there were only a few typos/grammar mistakes (listed later) that could be easily fixed.\n\n\n**Weaknesses and Concerns**\n\n- As a major concern, the regret analysis for NS-NAC is done under the very restrictive assumption that all policies induce uniformly ergodic Markov chains. Unless I am missing something, this essentially amounts to restricting to ergodic MDPs, which exclude many relevant domains in RL which are (weakly) communicating MDPs. If correct, this key fact must have been clearly stated earlier (e.g., Introduction) and must have been taken into account when positioning the algorithm relative to its model-based competitors that apply to wider classes (e.g., communicating). \n\n- Related to the previous concern, all the presented algorithms here crucially rely on the knowledge of $R_Q$, which itself depends on $\\lambda$, which might be related to mixing-type quantities. It's never discussed how crucial this dependence (i.e., knowing $R_Q$) is and how this can be interpreted. \n\n- Regarding NS-NAC: in line 5, why one would sample $s_t$ uniformly? Shouldn’t it be decided by the environment? \n\n- Introducing and analyzing the BORL variant of NS-NAC is nice. However, that it achieves the same regret bound --as the paper claims-- raises a key question as to whether the algorithm is parameter-free for free? Of course, the cost to pay could be multiplicative polylog-terms (in $T$) or numerical constants or even lower order terms. But this is neither discussed nor possible to understand from its regret analysis (see also the next comment). \n\n- The regret analysis of BORL-NS-NAC appears to be rushed: a final *clean* bound in missing in the proof (in p. 48). As mentioned above, a clean, explicit final regret bound (i.e., without hiding stuff in big-O notations) is especially crucial here to get insight into what is paid relative to the regret bound of NS-NAC. \n\n- Another concern (mostly presentational) is that the requirement of knowing $R_Q$ in BORL-NS-NAC must be made explicit in the introduction (and other relevant places).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The key statements are regret bounds as performance certificate of the presented algorithms, which are supported by the proofs that appear overall correct to me, even though I did not carefully check math details. \n\nIn terms of presentation, I raised some concerns regarding: \n- How the setting is advertised: the assumption of ergodicity is key and must have been emphasized earlier in the paper, especially when positioning the paper relative to the model-based competitors.\n- How the BORL-NS-NAC is advertised: it still depends on some input parameter, while the text induces this feeling that it is fully parameter-free.\n- Need to have a final, explicit regret bound for BORL-NS-NAC in its proof, which allows for comparison between NS-NAC and BORL-NS-NAC.", "audience_yn": "Yes", "audience_explanation": "The paper studies an important and interesting problem in the area of theoretical RL. It will attract TMLR readers because:\n- the considered setting (non-stationary RL in the average-reward setting) is technically challenging and interesting;  \n- the considered performance criterion (regret) is relevant to a wide-range of audience;\n- the developed algorithmic tools and analyses are interesting.", "requested_changes": "Please address the weaknesses mentioned above as well as my detailed comments under \"Additional Comments\".\n\nThe parameter-free extension of NS-NAC is indeed an important development from the standpoint of the method applicability, and I therefore think that it must be included in the main text, especially considering that there is some space for it.", "broader_impact_concerns": "The paper studies a theoretical problem, and its presented empirical evaluation is based on synthetic data. Therefore, I believe there is no concern associated to the developments and results reported in the paper.", "additional_comments": "**Some Detailed Comments and Questions**\n\n- Is your reward function deterministic? Please clarify. \n- As far as I see, the paper (41) is not related to NPG. Could you please elaborate further?\n- In Section 2, first paragraph: Note that the ref (7) studies the discounted setting. It does not render correct to include it in a discussion about average-reward setting. \n- I suggest citing some key papers for the assumptions used, especially for the structural results such as Equation (3).\n- Please provide a ref for the first few lines of Section 5.1, since they appear to come from the stationary PG analysis literature. \n- In Algorithm 1: The explanation for actor policy and $\\theta_{t+1}$ do not appear to be connected. Please revise accordingly. \n- In Corollary 5.4: What is condition number? What is recurrent coefficient?\n- When deriving the bound on Markovian noise: Isn’t correct that we could restrict to having $s_{t-\\tau-1}=\\tilde s_{t-\\tau-1}$? How such an adaptation is technically different from the stationary case?\n\n\n\n**Potential Typos/Grammar Mistakes, Notation Issues, and Unclear Sentences**\n\n- In Section 2, when introducing Advantage Actor-Critic, introduce the abbreviation A2C that is used later. \n- Use of this style (number) for citing a paper is misleading as one may confuse it easily with equation numbers. Why not [number]? Or (author, year)?\n- The abbreviation RL is readily introduced in the introduction, but then the authors switch to “Reinforcement Learning” (e.g., in Section 3.2, first line). \n- In the definition of $J^\\pi$, an expectation is missing. \n- What is “Theorem 5.1”. Did you mean “Assumption 5.1”? It is used in a number of places. \n- p. 4: The model of … and notion of … is standard  ==> … are standard \n- p. 4: In Bellman equation for $Q$, there is some unnecessary space before $=$. \n- p. 5: sub optimal ==> sub-optimal  \n- p. 8: i.e ==> i.e.\n- p. 5: Both $\\boldsymbol{\\pi}_{\\boldsymbol \\theta_t}$ and $\\boldsymbol{\\pi}_t$ are used to denote the same thing. \n- p. 8: … challenges that non-stationarity. ==> … challenges that non-stationarity poses OR … challenges due to non-stationarity. [Otherwise, the sentence appears incomplete.]\n- p. 9: time step ==> timestep [To ensure consistency with earlier uses.]\n- p. 9: $\\widetilde O_t$ is not defined. \n- p. 1: UCRL ==> UCRL2 (assuming that you are referring to the (meta-)algorithm of Jaksch et al., 2010)   \n- The paragraph “Function Approximation” in p. 7 seems to be redundant as it is already discussed in p. 6 (before beginning of Section 5).", "extra_scores": {}, "extra_text": {}}, {"review_id": "xO4hEaaLzS", "reviewer": "Reviewer_PnUT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present 2 model-free policy-based algorithms for non stationary MDPs under the average reward setting.\n- NS-NAC: a natural actor-critic algorithm that uses periodic restarts to handle the non stationary changes. However it requires to know the variation budgets.\n- BORL-NS-NAC: a relaxed version of the first algorithm that uses a bandit-over-RL framework to eliminate the need for prior knowledge of the environment's total variation budget.\n- Theoretical Analysis: a dynamic regret bound for both algorithms. The proof introduces a novel adaptation of Lyapunov analysis to a moving-target optimal policy.\n\nStrengths:\n- the paper fills a gap in the literature by providing the first theoretical analysis for a policy-based method in this setting\n- the inclusion of a relaxation (BORL-NS-NAC) enhances the practical relevance of the work.\n\nWeakness:\n- it looks that experimentally BORL-NS-NAC is not better than the baselines\n- there is a gap", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper's central claims are theoretical, and they are supported by a detailed proof. The analysis adapts standard, sound techniques to the novel problem setting. The empirical results on a synthetic environment, while not extensive, corroborate the theoretical findings by demonstrating sub-linear regret.", "audience_yn": "Yes", "audience_explanation": "The work addresses a fundamental problem in reinforcement learning: non-stationary MDP, which is of broad interest. It provides the first theoretical justification for using policy gradient methods, which are widely used in practice. Researchers in RL theory, continual learning, and policy optimization could find the results valuable.", "requested_changes": "In algorithm 1 and section 4, I can't find where ∆R,T and ∆P,T are needed. I do not think it is mentioned. Please make this clearer.", "broader_impact_concerns": "N/A", "additional_comments": "Policy gradient is usually more interesting than value based methods in continuous action spaces, can your analysis be easily extended to this setup?", "extra_scores": {}, "extra_text": {}}, {"review_id": "azVoyNV07b", "reviewer": "Reviewer_5B6M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis work lies in the field of non-stationary (NS) reinforcement learning (RL), considering the infinite-horizon average-reward setting.  \nThe authors claim to propose the first policy-based method for this setting, namely NS-NAC (non-stationary natural actor critic), which is a policy gradient method incorporating restart-based exploration for change. The latter means that the method divides the time horizon $T$ into $N$ segments of equal length at the beginning of which each NAC sub-routine is re-initialized. The authors show that selecting the step sizes as a function of the variation budget $\\Delta_{T}$ and $N=\\Delta_{T}^{5/6} T^{1/6}$, then NS-NAC suffers a dynamic regret of order $\\widetilde{\\mathcal{O}}(|\\mathcal{S}|^{1/2} |\\mathcal{A}|^{1/2} \\Delta_{T}^{1/6} T^{5/6})$.  While the results are presented for tabular softmax policies, they are extended to the case of function approximation in appendix.  \nFurthermore, the authors introduce a bandit over RL (BORL) variant of NS-NAC, namely BORL-NS-NAC, which deletes the need for passing $\\Delta_{T}$ as an input (required in NS-NAC to set the step sizes). Fundamentally, this variant runs the EXP3.P method over NS-NAC subroutines whose parameters are pulled by the overlying bandit method. The authors show that such a parameter-free variant suffers the same regret of the original version when the number of epochs is selected as $\\mathcal{O}(T^{2/3})$.  \nThe authors discuss their theoretical result against a lower bound for non-stationary undiscounted RL, conjecturing that the cause of the regret bound not to be tight is likely due to the analysis.  \nFinally, the proposed methods are validated in synthetic tabular settings in which they are compared against baselines, showing a sub-linear regret that aligns with the one exhibited by the competitors.\n\n**Strengths**\n1. The faced problem is relevant and, up to my knowledge, the proposed method(s) actually seems to be the first model-free policy-based method(s) for NS-RL. This opens to more practical usage of RL methods for NS environments, since policy-based methods come with notorious benefits for real-world-like problems.\n2. In Section 3 the authors introduce all the necessary knowledge to understand the content in the main paper, making the manuscript almost self-contained.\n3. Even if the authors state the main results for tabular environments, they provide an extension to function approximation\n4. The authors take the time to present their results and they extensively compare the guarantees of their method with the ones of other baselines (even if thought for different settings) and with the lower bound for the undiscounted setting.\n\n**Weaknesses**\n1. I feel that the BORL-NS-NAC method is not introduced in the best possible way. The method, which is meant to remove from NS-NAC the need of passing $\\Delta_{T}$ as input is not introduced, seems just an add-on in the abstract as well as in the original contribution. Moreover, when the reader gets to Section 6, it is very brief. Since I think that BORL-NS-NAC is an important extension of the main method (as also done in [12]), even because it presents the same guarantees, I suggest to introduce it properly, right from the beginning of the paper (abstract, original contribution, when presenting NS-NAC). To this end, the authors could stress deeply the importance of requiring $\\Delta_{T}$ as an input parameter, report the pseudo-code of the method in the main paper, comment on how the step sizes of the underlying subroutines are selected, comment why and how the guarantees are the same (indeed there is no commentary after Theorem 6.1). I suggest to bring back in the main paper many information that are deferred to Appendix F.\n2. Algorithm 1 does not show how to use $\\Delta_{T}$. I suggest to briefly specify how it is employed in the step-sizes. Moreover, it seems that NS-NAC does not need actually the knowledge of the budget to work and to show guarantees on the dynamic regret, but just to select the optimal step sizes to obtain the final bound for the regret.\n3. NS-NAC is based on fully restarting, meaning that after every amount of interaction, all the information collected are trashed, as commonly done in this setting [27]. However, do the authors believe that other approaches to forget the past (as sliding windows) may be well suited for this setting and possibly improving the results? \n4. In Section 5.2, when introducing the result on the universal regret, I feel that this part is not very self-contained. I think that the paper would benefit from more commentary on this part (from Appendix D.7), especially regarding the assumptions needed and their meaning, since at the moment the authors just briefly mention the condition number and recurrent coefficient in the statement of Corollary 5.4.\n5. The reported lower bound for the NS undiscounted setting from [16] lacks of the term $D^{2/3}$, being $D$ the maximum diameter of the MDPs. Is it just a typo, or is not important in this context (even if I guess it is)? Moreover, why the regret bounds for the method are not depending on the MDPs diameter? \n6. The regret bound are not tight w.r.t. the lower bound and the authors comment this by conjecturing (Section 5.2) that there is a slack in the analysis of the underlying NAC which is adapted from the stationary setting. Do the authors conjecture that a different analysis could lead to a tighter bound? Or is it a problem inside the method itself? What about starting from simpler on-policy policy gradient methods?\n7. I suggest to move Section 7 right after the presentation of the regret bound of NS-NAC.\n8. In practice, the proposed methods quite align with baselines, not showing a huge practical advantage, except for NS-NAC in the experiments in the appendix (Figure 4). Moreover, the validation section in the main paper does not describe the settings in which the methods are tested. Furthermore, BORL-NS-NAC is not considered in the appendix. Even if I think that the amount of experiments is enough for a theoretical paper, I would have appreciated a sensitivity study of NS-NAC to $\\Delta_{T}$ when compared to BORL-NS-NAC.\n\n**References**\nThe same of the paper under revision.", "claims_and_evidence_yn": "Yes", "claims_explanation": "For each theoretical result, the authors report the full proof if the result is new or a precise reference if the result is reported.\nThe method is also validated in synthetic settings, verifying that the methods show sublinear regret, aligning with the baselines.", "audience_yn": "Yes", "audience_explanation": "The paper seems to be the first to propose model-free policy-based method(s) for NS-RL, a complex setting that matches many real-world scenarios. This works opens to more practical usage of RL methods for NS environments, since policy-based methods come with notorious benefits for real-world-like problems. Moreover, I think that this paper leaves some room for future work for policy-based methods in NS-RL.", "requested_changes": "**Critical Changes**\n1. I suggest to elevate and integrate the BORL-NS-NAC Variant. Please see Weakness 1.       \n2. I suggest to clarify the role of $\\Delta_T$ in Algorithm 1 in the case in which it has to be explicited. Please, see Weakness 2.\n3. I suggest to clarify the lower bound comparison, especially regarding the missing term for the MDP diameter. Please see Weakness 5\n4. I suggest to make the part of Section 5.2 on universal regret more self-contained (see Weakness 4). Alternatively, it can just be briefly mentioned and deferred to appendix.\n5. I suggest to add more commentary for the experimental validation reported in the main paper, at least with a description of the environments employed (see Weakness 8).\n\n**Recommended Changes**\n1. Discuss a little bit more on restarting mechanism and compare it against other method for facing non-stationarity (see Weakness 3).\n2. If it is the case, discuss whether tighter bound can be achieved by analyzing a simpler on-policy policy gradient method first (also as a conjecture, leaving it for future work), to isolate the slack introduced by the NAC analysis (see Weakness 6). \n3. Please consider moving Section 7 to appear right after presenting the regret bound (after Section 5, see Weakness 7).\n4. Please consider to add BORL-NS-NAC to the experiments in the appendix for a more complete validation, as well as considering to do a sensitivity analysis centered on the $\\Delta_{T}$ parameter for NS-NAC, and compare it with BORL-NS-NAC (see Weakness 8).", "broader_impact_concerns": "I have no concerns on ethical implications of the work.", "additional_comments": "**Minors**\n 1. In every statements (in the main paper and in the appendix) the word \"Theorem\" is used instead of the word \"Assumption\".\n 2. Lemma 5.2 does not seem to be the Lemma 2 of [47], but it is exactly Assumption 4.1 of [33]. If it is the case, consider to change it as an assumption too, otherwise, I kindly request to shed light on this fact.\n 3. Typo at the top of page 21: \"[...] and and [...]\".\n\n**References**\nThe same of the paper under revision.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)