NeurIPS 2025

Finite-Time Bounds for
Average-Reward Fitted Q-Iteration
Jongmin Lee
Seoul National University
Department of Mathematical Sciences
dlwhd2000@snu.ac.kr
Ernest K. Ryu
UCLA
Department of Mathematics
eryu@math.ucla.edu
Abstract
Although there is an extensive body of work characterizing the sample complexity
of discounted-return offline RL with function approximations, prior work on the
average-reward setting has received significantly less attention, and existing ap-
proaches rely on restrictive assumptions, such as ergodicity or linearity of the MDP.
In this work, we establish the first sample complexity results for average-reward
offline RL with function approximation for weakly communicating MDPs, a much
milder assumption. To this end, we introduce Anchored Fitted Q-Iteration, which
combines the standard Fitted Q-Iteration with an anchor mechanism. We show
that the anchor, which can be interpreted as a form of weight decay, is crucial
for enabling finite-time analysis in the average-reward setting. We also extend
our finite-time analysis to the setup where the dataset is generated from a single-
trajectory rather than IID transitions, again leveraging the anchor mechanism.
1
Introduction
The goal of offline Reinforcement Learning (RL) is to find a near-optimal policy using a precollected
dataset without any direct interaction with the environment. Characterizing the sample complexity
for finding an ϵ-optimal policy using function approximation under assumptions that the offline
data has sufficient coverage over the whole state-action space has been an active area of theoretical
RL research. However, more prior work focuses on the discounted cumulative reward setup, and
research on obtaining sample complexity in the average reward has been limited due to the absence
of the discount factor and the complexity of the Bellman equation. Specifically, all prior works with
function approximation rely on restrictive assumptions such as ergodicity or linearity of the MDP.
Alth...