SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also define the Bellman operator  $\mathcal{T}$  such that  $\forall f:S\times \mathcal{A}$
$$
\mathcal {T} f (s, a) = \mathbb {E} _ {s, a} [ R (s, a) ] + \mathbb {E} _ {s ^ {\prime} \sim P (s, a)} \max  _ {a ^ {\prime}} f \left(s ^ {\prime}, a ^ {\prime}\right), \forall (s, a) \in \mathcal {S} \times \mathcal {A}, \tag {H.4}
$$
where  $R(s,a)\in \Delta [0,1]$  represents a stochastic reward function.
H.2 Notations
- Feature covariance matrix  $\Sigma_{k;h}$ :
$$
\boldsymbol {\Sigma} _ {k; h} = \sum_ {\tau = 1} ^ {k} X _ {h} \left(f ^ {\tau}\right) \left(X _ {h} \left(f ^ {\tau}\right)\right) ^ {\top} + \lambda I \tag {H.5}
$$
- Matrix Norm Zanette et al. [63]: for a matrix  $\Sigma$ , the matrix norm  $\| \boldsymbol{u} \|_{\Sigma}$  is defined as:
$$
\left\| \boldsymbol {u} \right\| _ {\boldsymbol {\Sigma}} = \sqrt {\boldsymbol {u} \boldsymbol {\Sigma} \boldsymbol {u} ^ {\top}} \tag {H.6}
$$
- Weighted  $\ell^2$  norm: for a given distribution  $\beta \in \Delta(S \times \mathcal{A})$  and a function  $f: S \times \mathcal{A} \mapsto \mathbb{R}$ , we denote the weighted  $\ell^2$  norm as:
$$
\left\| f \right\| _ {2, \beta} ^ {2} := \sqrt {\mathbb {E} _ {(s , a) \sim \beta} f ^ {2} (s , a)} \tag {H.7}
$$
- A stochastic reward function  $R(s, a) \in \Delta([0, 1])$
- For each offline data distribution  $\nu = \{\nu_{0},\dots,\nu_{H - 1}\}$ , the offline data set at time step  $h$ $(\nu_{h})$  contains data samples  $(s,a,r,s^{\prime})$ , where  $(s,a)\sim \nu_h,r\in R(s,a),s'\sim P(s,a)$ .
- Given a policy  $\pi \coloneqq \{\pi_0, \dots, \pi_{H - 1}\}$ , where  $\pi_h: S \mapsto \Delta(\mathcal{A})$ ,  $d_h^\pi \in \Delta(s, a)$  denotes the state-action occupancy induced by  $\pi$  at step  $h$ .
- We consider the value-based function approximation setting, where we are given a function class  $\mathcal{C} = \mathcal{C}_0 \times \ldots \mathcal{C}_{H - 1}$  with  $\mathcal{C}_h \subset \mathcal{S} \times \mathcal{A} \mapsto [0, V_{\max}]$ .
- A policy  $\pi^f$  is defined as the greedy policy w.r.t.  $f\colon \pi_h^f (s) = \arg \max_a f_h(s,a)$ . Specifically, at iteration  $k$ , we use  $\pi^k$  to denote the greedy policy w.r.t.  $f^k$ .
H.3 Assumptions and Definitions
Assumption H.1 (Pessimistic Realizability and Completeness). For any policy  $\pi^e$ , we say  $\mathcal{C}_h$  is a pessimistic function class w.r.t.  $\pi^e$ , if for any  $h$ , we have  $Q_h^{\pi^e} \in \mathcal{C}_h$ , and additionally, for any  $f_{h+1} \in \mathcal{C}_{h+1}$ , we have  $\mathcal{T}f_{h+1} \in \mathcal{C}_h$  and  $f_h(s,a) \leq Q_h^{\pi^e}(s,a), \forall (s,a) \in S \times \mathcal{A}$ .
Definition H.2 (Bilinear model Du et al. [9]). We say that the MDP together with the function class  $\mathcal{F}$  is a bilinear model of rand  $d$  of for any  $h\in [H - 1]$ , there exist two (known) mappings  $X_{h},W_{h}:\mathcal{F}\mapsto \mathbb{R}^{d}$  with  $\max_f\| X_h(f)\| _2\leq B_X$  and  $\max_f\| W_h(f)\| _2\leq B_W$  such that
$$
\forall f, g \in \mathcal {F}: \left| \mathbb {E} _ {s, a \sim d _ {h} ^ {\pi^ {f}}} [ g _ {h} (s, a) - \mathcal {T} g _ {h + 1} (s, a) ] \right| = | \langle X _ {h} (f), W _ {h} (g) \rangle |. \tag {H.8}
$$
Assumption H.3 (Bilinear Rank of Reference Policies). Suppose  $Q^{\mathrm{ref}} \in \mathcal{C}_{\mathrm{ref}} \subset \mathcal{C}$ , where  $\mathcal{C}_{\mathrm{ref}}$  is the function class of our reference policy, we assume the Bilinear rank of  $\mathcal{C}_{\mathrm{ref}}$  is  $d_{\mathrm{ref}}$  and  $d_{\mathrm{ref}} \leq d$ .
Definition H.4 (Calibrated Bellman error transfer coefficient). For any policy  $\pi$ , we define the calibrated transfer coefficient w.r.t. to a reference policy  $\pi^{\mathrm{ref}}$  as
$$
C _ {\pi} ^ {\text {r e f}} := \max  _ {f \in \mathcal {C}, f (s, a) \geq Q ^ {\text {r e f}} (s, a)} \frac {\sum_ {h = 0} ^ {H - 1} \mathbb {E} _ {s , a \sim d _ {h} ^ {\pi}} \left[ \mathcal {T} f _ {h + 1} (s , a) - f _ {h} (s , a) \right]}{\sqrt {\sum_ {h = 0} ^ {H - 1} \mathbb {E} _ {s , a \sim v _ {h}} \left(\mathcal {T} f _ {h + 1} (s , a) - f _ {h} (s , a)\right) ^ {2}}}, \tag {H.9}
$$
where  $Q^{\mathrm{ref}} = Q^{\pi^{\mathrm{ref}}}$
H.4 Discussions on the Assumptions
The pessimistic realizability and completeness assumption (Assumption H.1) is motivated by some theoretical studies of the pessimistic offline methods [59, 6] with regularizers:
$$
\min  _ {\theta} \alpha \underbrace {\left(\mathbb {E} _ {s \sim \mathcal {D} , a \sim \pi} \left[ Q _ {\theta} (s , a) \right] - \mathbb {E} _ {s , a \sim \mathcal {D}} \left[ Q _ {\theta} (s , a) \right]\right)} _ {\text {C o n s e r v a t i v e r e g u l a r i z e r} \mathcal {R} (\theta)} + \frac {1}{2} \mathbb {E} _ {s, a, s ^ {\prime} \sim \mathcal {D}} \left[ \left(Q _ {\theta} (s, a) - \mathcal {B} ^ {\pi} \bar {Q} (s, a)\right) ^ {2} \right]. \tag {H.10}
$$
Since the goal of the conservative regularizer  $\mathcal{R}(\theta)$  intrinsically wants to enforce
$$
Q _ {\theta} (s, \pi (s)) \leq Q _ {\theta} (s, \pi^ {e} (s)), \tag {H.11}
$$
where  $\pi$  is the training policy and  $\pi^e$  is the reference (behavior) policy. One can consider (H.10) as the Lagrange duality formulation of the following primal optimization problem:
$$
\min  _ {\theta} \mathbb {E} _ {s, a, s ^ {\prime} \sim \mathcal {D}} \left[ \left(Q _ {\theta} (s, a) - \mathcal {B} ^ {\pi} \bar {Q} (s, a)\right) ^ {2} \right], \text {s u b j e c t} \mathbb {E} _ {s \sim \mathcal {D}, a \sim \pi} \left[ Q _ {\theta} (s, a) \right] \leq \mathbb {E} _ {s \sim \mathcal {D}, a \sim \pi^ {e}} \left[ Q _ {\theta} (s, a) \right], \tag {H.12}
$$
where the constraint set is equivalent to Assumption H.1. Although Assumption H.1 directly characterizes the constraint set of the primal form of (H.10) the exact theoretical connection between the pessimistic realizability and the regularized bellman consistency equation is beyond the scope of this work and we would like to leave that for future studies.
Assumption H.1 allows us to restrict the functional class of interest to a smaller conservative function class  $\mathcal{C} \subset \mathcal{F}$ , which leads to a smaller Bellman rank of the reference policy ( $d_{\mathrm{ref}} \leq d$ ) suggested in Assumption H.3, and a smaller concentrability coefficient ( $C_{\pi}^{\mathrm{ref}} \leq C_{\pi}$ ) defined in Definition H.4, and I.2. Assumption H.3 and Definition I.2 provide the Bellman Bilinear rank and Bellman error transfer coefficient of the pessimistic functional class  $\mathcal{C}$  of interest.
H.5 Proof Structure Overview
We provide an overview of the proof structure and its dependency on different assumptions below:
- Theorem H.5: the total regret is decomposed into offline regrets and online regrets.
- Bounding offline regrets, requiring Definition H.4 and the following lemmas:
* Performance difference lemma w.r.t. a comparator policy (Lemma 1.5).
* Least square generalization bound (Lemma I.4), requiring Assumption H.1.
- Bounding online regrets, requiring Definition H.2
* Performance difference lemma for the online error (Lemma I.6).
* Least square generalization bound (Lemma I.4), requiring Assumption H.1.
* Upper bounds with the bilinear model assumption (Lemma 1.7).
* Applying Elliptical Potential Lemma [35] with bellman rank  $d$  and  $d_{\mathrm{ref}}$  (Lemma I.8), requiring Assumption H.3.
H.6 Our Results
Theorem H.5 (Formal Result of Theorem 6.1). Fix  $\delta \in (0,1)$ ,  $m_{\mathrm{off}} = K$ ,  $m_{\mathrm{on}} = 1$ , suppose and the function class  $\mathcal{C}$  follows Assumption H.1 w.r.t.  $\pi^e$ . Suppose the underlying MDP admits Bilinear rank  $d$  on function class  $\mathcal{C}$  and  $d_{\mathrm{ref}}$  on  $\mathcal{C}_{\mathrm{ref}}$ , respectively, then with probability at least  $1 - \delta$ , Algorithm 2 obtains the following bound on cumulative suboptimality w.r.t. any comparator policy  $\pi^e$ :
$$
\sum_ {t = 1} ^ {K} V ^ {\pi^ {e}} - V ^ {\pi^ {k}} = \widetilde {O} \left(\min  \left\{C _ {\pi^ {e}} ^ {\text {r e f}} H \sqrt {d K \log (| \mathcal {F} | / \delta)}, K \left(V ^ {\pi^ {e}} - V ^ {\text {r e f}}\right) + H \sqrt {d _ {\text {r e f}} K \log (| \mathcal {F} | / \delta)} \right\}\right). \tag {H.13}
$$
Note that Theorem H.5 provides a guarantee for any comparator policy  $\pi^{\mathrm{e}}$ , which can be directly applied to  $\pi^{\star}$  described in our informal result (Theorem 6.1). We also change the notation for the reference policy from  $\mu$  in Theorem 6.1 to  $\pi^{\mathrm{ref}}$  (similarly,  $d_{\mathrm{ref}}, V^{\mathrm{ref}}, C_{\pi^e}^{\mathrm{ref}}$  correspond to  $d_{\mu}, V^{\mu}, C_{\pi^e}^{\mu}$  in Theorem 6.1) for notation consistency in the proof. Our proof of Theorem H.5 largely follows the proof of Theorem 1 of [52], and the major changes are caused by Assumption H.1, Assumption H.3, and Definition H.4.
Proof. Let  $\mathcal{E}_k$  denote the event that  $\left\{f_0^k (s,a)\leq Q^{\mathrm{ref}}(s,a)\right\}$  and  $\bar{\mathcal{E}}_k$  denote the event that  $\left\{f_0^k (s,a) > Q^{\mathrm{ref}}(s,a)\right\}$ . Let  $V^{\mathrm{ref}}(s) = \max_{a}Q^{\mathrm{ref}}(s,a)$ , we start by noting that
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} V ^ {\pi^ {e}} - V ^ {\pi^ {f ^ {k}}} = \sum_ {k = 1} ^ {K} \mathbb {E} _ {s \sim \rho} \left[ V _ {0} ^ {\pi^ {e}} (s) - V _ {0} ^ {\pi^ {f ^ {k}}} (s) \right] \\ = \underbrace {\sum_ {k = 1} ^ {K} \mathbb {E} _ {s \sim \rho} \left[ \mathbb {1} \left\{\bar {\mathcal {E}} _ {k} \right\} \left(V _ {0} ^ {\pi^ {e}} (s) - V ^ {\mathrm {r e f}} (s)\right) \right]} _ {\Gamma_ {0}} + \underbrace {\sum_ {k = 1} ^ {K} \mathbb {E} _ {s \sim \rho} \left[ \mathbb {1} \left\{\bar {\mathcal {E}} _ {k} \right\} \left(V ^ {\mathrm {r e f}} (s) - \max  _ {a} f _ {0} ^ {k} (s , a)\right) \right]} _ {= 0, \text {b y t h e d e f i n i t i o n o f} \bar {\mathcal {E}} _ {k} \text {a n d l i n e 9 o f A l g o r i t h m 2}} \\ + \underbrace {\sum_ {t = 1} ^ {K} \mathbb {E} _ {s \sim \rho} \left[ \mathbb {1} \left\{\bar {\mathcal {E}} _ {k} \right\} \left(\max _ {a} f _ {0} ^ {k} (s , a) - V _ {0} ^ {\pi^ {f ^ {k}}} (s)\right) \right]} _ {\Gamma_ {1}} + \underbrace {\sum_ {k = 1} ^ {K} \mathbb {E} _ {s \sim \rho} \left[ \mathbb {1} \left\{\mathcal {E} _ {k} \right\} \left(V _ {0} ^ {\pi^ {e}} (s) - \max _ {a} f _ {0} ^ {k} (s , a)\right) \right]} _ {\Gamma_ {2}} \\ + \underbrace {\sum_ {t = 1} ^ {T} \mathbb {E} _ {s \sim \rho} \left[ \mathbb {1} \left\{\mathcal {E} _ {k} \right\} \left(\max  _ {a} f _ {0} ^ {k} (s , a) - V _ {0} ^ {\pi^ {f ^ {k}}} (s)\right) \right]} _ {\Gamma_ {3}}. \tag {H.14} \\ \end{array}
$$
Let  $K_{1} = \sum_{k=1}^{K} \mathbb{1}\left\{f_{0}^{k}(s, a) > Q^{\mathrm{ref}}(s, a)\right\}$  and  $K_{2} = \sum_{k=1}^{K} \mathbb{1}\left\{f_{0}^{k}(s, a) \leq Q^{\mathrm{ref}}(s, a)\right\}$  (or equivalently  $K_{1} = \sum_{k=1}^{K} \mathbb{1}\left\{\bar{\mathcal{E}}_{k}\right\}, K_{2} = \sum_{k=1}^{K} \mathbb{1}\left\{\mathcal{E}_{k}\right\}$ ). For  $\Gamma_{0}$ , we have
$$
\Gamma_ {0} = K _ {2} \mathbb {E} _ {s \sim \rho} \left(V ^ {\pi^ {e}} (s) - V ^ {\text {r e f}} (s)\right). \tag {H.15}
$$
For  $\Gamma_{2}$ , we have
$$
\begin{array}{l} \Gamma_ {2} = \sum_ {k = 1} ^ {K} \mathbb {E} _ {s \sim \rho} \left[ \mathbb {1} \left\{\mathcal {E} _ {k} \right\} \left(V _ {0} ^ {\pi^ {e}} (s) - \max  _ {a} f _ {0} ^ {k} (s, a)\right) \right] \\ \leq \sum_ {k = 1} ^ {(i)} K \mathbb {1} \left\{\mathcal {E} _ {k} \right\} \sum_ {h = 0} ^ {H - 1} \mathbb {E} _ {s, a \sim d _ {h} ^ {r ^ {e}}} \left[ \mathcal {T} f _ {h + 1} ^ {k} (s, a) - f _ {h} ^ {k} (s, a) \right] \tag {H.16} \\ \stackrel {(i i)} {\leq} \sum_ {k = 1} ^ {K} \left[ C _ {\pi^ {e}} ^ {\text {r e f}} \cdot \mathbb {1} \left\{\mathcal {E} _ {k} \right\} \sqrt {\sum_ {h = 0} ^ {H - 1} \mathbb {E} _ {s , a \sim \nu_ {h}} \left[ \left(f _ {h} ^ {k} (s , a) - \mathcal {T} f _ {h + 1} ^ {k} (s , a)\right) ^ {2} \right]} \right] \\ \stackrel {(i i i)} {\leq} K _ {1} C _ {\pi^ {e}} ^ {\text {r e f}} \sqrt {H \cdot \Delta_ {\text {o f f}}}, \\ \end{array}
$$
where  $\Delta_{\mathrm{off}}$  is similarly defined as Song et al. [52] (See (I.3) of Lemma I.4). Inequality  $(i)$  holds because of Lemma I.5, inequality  $(ii)$  holds by the definition of  $C_{\pi^e}^{\mathrm{ref}}$  (Definition H.4), inequality  $(iii)$  holds by applying Lemma I.4 with the function class satisfying Assumption H.1, and Definition H.4. Note that the telescoping decomposition technique in the above equation also appears in [58, 24, 9]. Next, we will bound  $\Gamma_1 + \Gamma_3$ :
$$
\begin{array}{l} \Gamma_ {1} + \Gamma_ {3} = \sum_ {k = 1} ^ {K} \left(\mathbb {1} \left\{\mathcal {E} _ {k} \right\} + \mathbb {1} \left\{\bar {\mathcal {E}} _ {k} \right\}\right) \mathbb {E} _ {s \sim d _ {0}} \left[ \max  _ {a} f _ {0} ^ {k} (s, a) - V _ {0} ^ {\pi^ {f ^ {k}}} (s) \right] \\ \stackrel {(i)} {\leq} \sum_ {k = 1} ^ {K} \left(\mathbb {1} \left\{\mathcal {E} _ {k} \right\} + \mathbb {1} \left\{\bar {\mathcal {E}} _ {k} \right\}\right) \sum_ {h = 0} ^ {H - 1} \left| \mathbb {E} _ {s, a \sim d _ {h} ^ {\pi^ {f ^ {k}}}} \left[ f _ {h} ^ {k} (s, a) - \mathcal {T} f _ {h + 1} ^ {k} (s, a) \right] \right| \tag {H.17} \\ \stackrel {(i i)} {=} \sum_ {t = 1} ^ {K} \left[ \left(\mathbb {1} \left\{\mathcal {E} _ {k} \right\} + \mathbb {1} \left\{\bar {\mathcal {E}} _ {k} \right\}\right) \sum_ {h = 0} ^ {H - 1} | \langle X _ {h} (f ^ {k}), W _ {h} (f ^ {k}) \rangle | \right] \\ \stackrel {(i i i)} {\leq} \sum_ {k = 1} ^ {K} \left[ \left(\mathbb {1} \left\{\mathcal {E} _ {k} \right\} + \mathbb {1} \left\{\bar {\mathcal {E}} _ {k} \right\}\right) \sum_ {h = 0} ^ {H - 1} \left\| X _ {h} (f ^ {k}) \right\| _ {\boldsymbol {\Sigma} _ {k - 1; h} ^ {- 1}} \sqrt {\Delta_ {\mathrm {o n}} + \lambda B _ {W} ^ {2}} \right], \\ \end{array}
$$
where  $\Delta_{\mathrm{on}}$  is similarly defined as Song et al. [52] (See (I.4) of Lemma I.4). Inequality  $(i)$  holds by Lemma I.6, equation  $(ii)$  holds by the definition of Bilinear model ((H.8) in Definition H.2), inequality  $(iii)$  holds by Lemma I.7 and Lemma I.4 with the function class satisfying Assumption H.1. Using Lemma I.8, we have that
$$
\begin{array}{l} \Gamma_ {1} + \Gamma_ {3} \\ \leq \sum_ {k = 1} ^ {K} \left[ \left(\mathbb {1} \left\{\mathcal {E} _ {k} \right\} + \mathbb {1} \left\{\bar {\mathcal {E}} _ {k} \right\}\right) \sum_ {h = 0} ^ {H - 1} \left\| X _ {h} \left(f ^ {k}\right) \right\| _ {\boldsymbol {\Sigma} _ {k - 1; h} ^ {- 1}} \sqrt {\Delta_ {\mathrm {o n}} + \lambda B _ {W} ^ {2}} \right] \\ \stackrel {(i)} {\leq} H \sqrt {2 d \log \left(1 + \frac {K _ {1} B _ {X} ^ {2}}{\lambda d}\right) \cdot \left(\Delta_ {\mathrm {o n}} + \lambda B _ {W} ^ {2}\right) \cdot K _ {1}} + H \sqrt {2 d _ {\mathrm {r e f}} \log \left(1 + \frac {K _ {2} B _ {X} ^ {2}}{\lambda d _ {\mathrm {r e f}}}\right) \cdot \left(\Delta_ {\mathrm {o n}} + \lambda B _ {W} ^ {2}\right) \cdot K _ {2}}} \\ \stackrel {(i i)} {\leq} H \left(\sqrt {2 d \log \left(1 + \frac {K _ {1}}{d}\right) \cdot \left(\Delta_ {\text {o n}} + B _ {X} ^ {2} B _ {W} ^ {2}\right) \cdot K _ {1}} + \sqrt {2 d _ {\text {r e f}} \log \left(1 + \frac {K _ {2}}{d _ {\text {r e f}}}\right) \cdot \left(\Delta_ {\text {o n}} + B _ {X} ^ {2} B _ {W} ^ {2}\right) \cdot K _ {2}}\right), \tag {H.18} \\ \end{array}
$$
where the first part of inequality  $(i)$  holds by the assumption that the underlying MDPs have bellman rank  $d$  (Definition H.2) when  $\bar{\mathcal{E}}_k$  happens, and the second part of inequality  $(i)$  holds by the assumption that  $\mathcal{C}_{\mathrm{ref}}$  has bilinear rank  $d_{\mathrm{ref}}$  (Assumption H.3)  $\mathcal{C}_{\mathrm{ref}}$  has bellman rank  $d_{\mathrm{ref}}$  when  $\mathcal{E}_k$  happens. Inequality  $(ii)$  holds by plugging in  $\lambda = B_X^2$ . Substituting (H.15), inequality H.16, and inequality (H.18) into (H.14), we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {K} V ^ {\pi^ {e}} - V ^ {\pi^ {f ^ {k}}} \leq \Gamma_ {0} + \Gamma_ {2} + \Gamma_ {1} + \Gamma_ {3} \leq K _ {2} \left(V ^ {\pi^ {e}} (s) - V ^ {\mathrm {r e f}} (s)\right) + K _ {1} C _ {\pi^ {e}} ^ {\mathrm {r e f}} \sqrt {H \cdot \Delta_ {\mathrm {o f f}}} \\ + H \left(\sqrt {2 d \log \left(1 + \frac {K _ {1}}{d}\right) \cdot \left(\Delta_ {\text {o n}} + B _ {X} ^ {2} B _ {W} ^ {2}\right) \cdot K _ {1}} + \sqrt {2 d _ {\text {r e f}} \log \left(1 + \frac {K _ {2}}{d _ {\text {r e f}}}\right) \cdot \left(\Delta_ {\text {o n}} + B _ {X} ^ {2} B _ {W} ^ {2}\right) \cdot K _ {2}}\right) \tag {H.19} \\ \end{array}
$$
Plugging in the values of  $\Delta_{\mathrm{on}}$ ,  $\Delta_{\mathrm{off}}$  from (I.3) and (I.4), and using the subadditivity of the square root function, we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} V ^ {\pi^ {e}} - V ^ {\pi^ {f ^ {k}}} \\ \leq K _ {2} \left(V ^ {\pi^ {e}} (s) - V ^ {\text {r e f}} (s)\right) + 1 6 V _ {\max } C _ {\pi^ {e}} ^ {\text {r e f}} K _ {1} \sqrt {\frac {H}{m _ {\text {o f f}}} \log \left(\frac {2 H K _ {1} | \mathcal {F} |}{\delta}\right)} \tag {H.20} \\ + \left(1 6 V _ {\max } \sqrt {\frac {1}{m _ {\mathrm {o n}}} \log \left(\frac {2 H K _ {1} | \mathcal {F} |}{\delta}\right)} + B _ {X} B _ {W}\right) \cdot H \sqrt {2 d K _ {1} \log \left(1 + \frac {K _ {1}}{d}\right)} \\ + \left(1 6 V _ {\max } \sqrt {\frac {1}{m _ {\mathrm {o n}}} \log \left(\frac {2 H K _ {2} | \mathcal {F} |}{\delta}\right)} + B _ {X} B _ {W}\right) \cdot H \sqrt {2 d _ {\mathrm {r e f}} K _ {2} \log \left(1 + \frac {K _ {2}}{d _ {\mathrm {r e f}}}\right)}. \\ \end{array}
$$
Setting  $m_{\mathrm{off}} = K$ ,  $m_{\mathrm{on}} = 1$  in the above equation completes the proof, we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} V ^ {\pi^ {e}} - V ^ {\pi^ {k}} \\ \leq \widetilde {O} \left(C _ {\pi^ {e}} ^ {\text {r e f}} \sqrt {H K _ {1} \log (| \mathcal {F} | / \delta)}\right) + \widetilde {O} \left(H \sqrt {d K _ {1} \log (| \mathcal {F} | / \delta)}\right) \\ + K _ {2} \left(V ^ {\pi^ {e}} (s) - V ^ {\text {r e f}} (s)\right) + \widetilde {O} \left(H \sqrt {d _ {\text {r e f}} K _ {2} \log \left(| \mathcal {F} | / \delta\right)}\right) \tag {H.21} \\ \end{array}
$$
$$
\begin{array}{l} \leq \left\{ \begin{array}{l l} \widetilde {O} \left(C _ {\pi^ {e}} ^ {\text {r e f}} H \sqrt {d K _ {1} \log (| \mathcal {F} | / \delta)}\right) & \text {i f K _ {1} \gg K _ {2}}, \\ \widetilde {O} \left(K _ {2} \left(V ^ {\pi^ {e}} - V ^ {\text {r e f}}\right) + H \sqrt {d _ {\text {r e f}} K _ {2} \log (| \mathcal {F} | / \delta)}\right) & \text {o t h e r w i s e}. \end{array} \right. \\ \leq \widetilde {O} \left(\min  \left\{C _ {\pi^ {e}} ^ {\text {r e f}} H \sqrt {d K \log (| \mathcal {F} | / \delta)}, K \left(V ^ {\pi^ {e}} - V ^ {\text {r e f}}\right) + H \sqrt {d _ {\text {r e f}} K \log (| \mathcal {F} | / \delta)} \right\}\right), \\ \end{array}
$$
where the last inequality holds because  $K_{1}, K_{2} \leq K$ , which completes the proof.
I Key Results of HyQ [52]
In this section, we restate the major theoretical results of Hy-Q [52] for completeness.
I.1 Assumptions
Assumption I.1 (Realizability and Bellman completeness). For any  $h$ , we have  $Q_h^\star \in \mathcal{F}_h$ , and additionally, for any  $f_{h+1} \in \mathcal{F}_{h+1}$ , we have  $\mathcal{T}f_{h+1} \in \mathcal{F}_h$ .
Definition I.2 (Bellman error transfer coefficient). For any policy  $\pi$ , we define the transfer coefficient as
$$
C _ {\pi} := \max  \left\{0, \max  _ {f \in \mathcal {F}} \frac {\sum_ {h = 0} ^ {H - 1} \mathbb {E} _ {s , a \sim d _ {h} ^ {\pi}} \left[ \mathcal {T} f _ {h + 1} (s , a) - f _ {h} (s , a) \right]}{\sqrt {\sum_ {h = 0} ^ {H - 1} \mathbb {E} _ {s , a \sim \nu_ {h}} \left(\mathcal {T} f _ {h + 1} (s , a) - f _ {h} (s , a)\right) ^ {2}}} \right\}. \tag {I.1}
$$
I.2 Main Theorem of Hy-Q
Theorem I.3 (Theorem 1 of Song et al. [52]). Fix  $\delta \in (0,1)$ ,  $m_{\mathrm{off}} = K$ ,  $m_{\mathrm{on}} = 1$ , and suppose that the underlying MDP admits Bilinear rank  $d$  (Definition H.2), and the function class  $\mathcal{F}$  satisfies Assumption I.1. Then with probability at least  $1 - \delta$ , HyQ obtains the following bound on cumulative suboptimality w.r.t. any comparator policy  $\pi^e$ :
$$
\operatorname {R e g} (K) = \widetilde {O} \left(\max  \left\{C _ {\pi^ {e}}, 1 \right\} V _ {\max } B _ {X} B _ {W} \sqrt {d H ^ {2} K \cdot \log \left(| \mathcal {F} | / \delta\right)}\right). \tag {I.2}
$$
I.3 Key Lemmas
I.3.1 Least Squares Generalization and Applications
Lemma I.4 (Lemma 7 of Song et al. [52], Online and Offline Bellman Error Bound for FQI). Let  $\delta \in (0,1)$  and  $\forall h\in [H - 1],k\in [K]$ , let  $f_h^{k + 1}$  be the estimated value function for time step  $h$
computed via least square regression using samples in the dataset  $\{\mathcal{D}_h^\nu, \mathcal{D}_h^1, \ldots, \mathcal{D}_h^T\}$  in (H.1) in the iteration  $t$  of Algorithm 2. Then with probability at least  $1 - \delta$ , for any  $h \in [H - 1]$  and  $k \in [K]$ , we have
$$
\left\| f _ {h} ^ {k + 1} - \mathcal {T} f _ {h + 1} ^ {k + 1} \right\| _ {2, \nu_ {h}} ^ {2} \leq \frac {1}{m _ {\text {o f f}}} 2 5 6 V _ {\max } ^ {2} \log (2 H K | \mathcal {F} | / \delta) =: \Delta_ {\text {o f f}} \tag {I.3}
$$
$$
\sum_ {\tau = 1} ^ {k} \left\| f _ {h} ^ {k + 1} - \mathcal {T} f _ {h + 1} ^ {k + 1} \right\| _ {2, \mu_ {h} ^ {\tau}} ^ {2} \leq \frac {1}{m _ {\mathrm {o n}}} 2 5 6 V _ {\max } ^ {2} \log (2 H K | \mathcal {F} | / \delta) =: \Delta_ {\mathrm {o n}}, \tag {I.4}
$$
where  $\nu_{h}$  denotes the offline data distribution at time  $h$ , and the distribution  $\mu_h^\tau \in \Delta (s,a)$  is defined such that  $s, a \sim d_h^{\pi^\tau}$ .
I.3.2 Bounding Offline Suboptimality via Performance Difference Lemma
Lemma 1.5 (Lemma 5 of Song et al. [52], performance difference lemma of w.r.t.  $\pi^e$ ). Let  $\pi^e = (\pi_0^e, \dots, \pi_{H-1}^e)$  be a comparator policy and consider any value function  $f = (f_0, \dots, f_{H-1})$ , where  $f_h: S \times \mathcal{A} \mapsto \mathbb{R}$ . Then we have
$$
\mathbb {E} _ {s \sim d _ {0}} \left[ V _ {0} ^ {\pi^ {e}} (s) - \max  _ {a} f _ {0} (s, a) \right] \leq \sum_ {i = 1} ^ {H - 1} \mathbb {E} _ {s, a \sim d _ {i} ^ {\pi^ {e}}} \left[ \mathcal {T} f _ {i + 1} (s, a) - f _ {i} (s, a) \right], \tag {I.5}
$$
where we define  $f_{H}(s,a) = 0,\forall (s,a)$
I.3.3 Bounding Online Suboptimality via Performance Difference Lemma
Lemma 1.6 (Lemma 4 of Song et al. [52], performance difference lemma). For any function  $f = (f_0, \ldots, f_{H-1})$  where  $f_h : \mathcal{S} \times \mathcal{A} \mapsto \mathbb{R}$  and  $h \in [H-1]$ , we have
$$
\mathbb {E} _ {s \sim d _ {0}} \left[ \max  _ {a} f _ {0} (s, a) - V _ {0} ^ {\pi^ {f}} (s) \right] \leq \sum_ {h = 0} ^ {H - 1} \left| \mathbb {E} _ {s, a \sim d _ {h} ^ {\pi^ {f}}} [ f _ {h} (s, a) - \mathcal {T} f _ {h + 1} (s, a) ] \right|, \tag {I.6}
$$
where we define  $f_{H}(s,a) = 0,\forall s,a$
Lemma 1.7 (Lemma 8 of Song et al. [52], upper bounding bilinear class). For any  $k \geq 2$  and  $h \in [H - 1]$ , we have
$$
\left| \left\langle W _ {h} \left(f ^ {k}\right), X _ {h} \left(f ^ {k}\right) \right\rangle \right| \leq \left\| X _ {h} \left(f ^ {k}\right) \right\| _ {\boldsymbol {\Sigma} _ {k - 1; h} ^ {- 1}} \sqrt {\sum_ {i = 1} ^ {k - 1} \mathbb {E} _ {s , a \sim d _ {h} ^ {f ^ {i}}} \left[ \left(f _ {h} ^ {k} - \mathcal {T} f _ {h + 1} ^ {k}\right) ^ {2} \right] + \lambda B _ {W} ^ {2}}, \tag {I.7}
$$
where  $\pmb{\Sigma}_{k-1;h}$  is defined as (H.5) and we use  $d_h^{f^k}$  to denote  $d_h^{\pi^{f^k}}$ .
Lemma 1.8 (Lemma 6 of Song et al. [52], bounding the inverse covariance norm). Let  $X_{h}(f^{1}),\ldots ,X_{h}(f^{K})\in \mathbb{R}^{d}$  be a sequence of vectors with  $\| X_h(f^k)\| _2\leq B_X < \infty ,\forall k\leq K$  Then we have
$$
\sum_ {k = 1} ^ {K} \left\| X _ {h} \left(f ^ {k}\right) \right\| _ {\boldsymbol {\Sigma} _ {k - 1; h} ^ {- 1}} \leq \sqrt {2 d K \log \left(1 + \frac {K B _ {X} ^ {2}}{\lambda d}\right)}, \tag {I.8}
$$
where we define  $\pmb{\Sigma}_{k;h} := \sum_{\tau=1}^{k} X_h(f^\tau) X_h(f^\tau)^T + \lambda \pmb{I}$  and we assume  $\lambda \geq B_X^2$  holds  $\forall k \in [K]$ .