SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma D.4 (Concentration for KL Divergence, Proposition 1 in [Jonsson et al., 2020]). Let  $X_{1}, X_{2}, \ldots, X_{n}, \ldots$  be i.i.d. samples from a distribution supported over  $\{1, \ldots, m\}$ , of probabilities given by  $\mathbb{P} \in \Sigma_{m}$ , where  $\Sigma_{m}$  is the probability simplex of dimension  $m - 1$ . We denote by  $\widehat{\mathbb{P}}_n$  the empirical vector of probabilities. Then, for any  $\delta \in [0,1]$ , with probability at least  $1 - \delta$ , it holds that
$$
\forall n \in \mathbb {N}, \operatorname {K L} \left(\widehat {\mathbb {P}} _ {n}, \mathbb {P}\right) \leq \frac {1}{n} \log \left(\frac {1}{\delta}\right) + \frac {m}{n} \log \left(e \left(1 + \frac {n}{m}\right)\right).
$$
We remark that there is a slight difference between it and the original version. In [Jonsson et al., 2020], they use  $m - 1$  instead of  $m$ . But since the second term of the right hand side above is increasing with  $m$ , our version also holds.
Lemma D.5 (Bernstein Transportation, Lemma 11 in [Talebi and Maillard, 2018]). For any function  $f$  and any two probability measures  $\mathbb{Q}, \mathbb{P}$  which satisfy  $\mathbb{Q} \ll \mathbb{P}$ , we denote  $\mathbb{V}_P[f] := \operatorname{Var}_{X \sim \mathbb{P}}(f(X))$  and  $\mathbb{S}(f) := \sup_x f(x) - \inf_x f(x)$ . We assume  $\mathbb{V}_P[f]$  and  $\mathbb{S}(f)$  are finite, then we have
$$
\mathbb {E} _ {Q} [ f ] - \mathbb {E} _ {P} [ f ] \leqslant \sqrt {2 \mathbb {V} _ {P} [ f ] \mathrm {K L} (Q , P)} + \frac {2}{3} \mathbb {S} (f) \mathrm {K L} (Q, P),
$$
$$
\mathbb {E} _ {P} [ f ] - \mathbb {E} _ {Q} [ f ] \leqslant \sqrt {2 \mathbb {V} _ {P} [ f ] \mathrm {K L} (Q , P)}.
$$
Lemma D.6 (Difference of Variance, Lemma 12 in [Ménard et al., 2021]). Let  $\mathbb{P}$ ,  $\mathbb{Q}$  be two probability measure on a discrete sample space of cardinality  $S$ . Let  $f, g$  be two functions defined on  $S$  such that  $0 \leq g(s), f(s) \leq b$  for all  $s \in S$ , we have that
$$
\operatorname {V a r} _ {\mathbb {P}} (f) \leq 2 \operatorname {V a r} _ {\mathbb {P}} (g) + 2 b \mathbb {E} _ {\mathbb {P}} | f - g | \quad a n d
$$
$$
\operatorname {V a r} _ {\mathbb {Q}} (f) \leq \operatorname {V a r} _ {\mathbb {Q}} (f) + 3 b ^ {2} \| \mathbb {P} - \mathbb {Q} \| _ {1},
$$
Further, if  $\mathrm{KL}(\mathbb{P};\mathbb{Q})\leq \alpha$  , it holds that
$$
\operatorname {V a r} _ {\mathbb {Q}} (f) \leq 2 \operatorname {V a r} _ {\mathbb {P}} (f) + 4 b ^ {2} \alpha .
$$
Lemma D.7. For any sequence of numbers  $z_{1},\ldots ,z_{n}$  with  $0\leq z_{k}\leq 1$  , we have
$$
\sum_ {k = 1} ^ {n} \frac {z _ {k}}{\max \left[ 1 ; \sum_ {i = 1} ^ {k - 1} z _ {i} \right]} \leq 4 \log \left(\sum_ {i = 1} ^ {n} z _ {i} + 1\right)
$$
$$
\begin{array}{l} \sum_ {k = 1} ^ {n} \frac {z _ {k}}{\max  \left[ 1 ; \sum_ {i = 1} ^ {k - 1} z _ {i} \right]} \leq 4 \sum_ {k = 1} ^ {n} \frac {\sum_ {i = 1} ^ {k} z _ {i} - \sum_ {i = 1} ^ {k - 1} z _ {i}}{2 + 2 \sum_ {i = 1} ^ {k - 1} z _ {i}} \\ \leq 4 \sum_ {k = 1} ^ {n} \frac {\sum_ {i = 1} ^ {k} z _ {i} - \sum_ {i = 1} ^ {k - 1} z _ {i}}{1 + \sum_ {i = 1} ^ {k} z _ {i}} \\ \leq 4 \sum_ {k = 1} ^ {n} \int_ {\sum_ {i = 1} ^ {k - 1} z _ {i}} ^ {\sum_ {i = 1} ^ {k} z _ {i}} \frac {1}{1 + x} d x \leq 4 \log \left(\sum_ {i = 1} ^ {n} z _ {i} + 1\right). \\ \end{array}
$$
Lemma D.8. For  $B \geq 16$  and  $x \geq 3$ , there exists a universal constant  $C_1 \geq 4$ , such that when
$$
x \geq C _ {1} B \log (B) ^ {2},
$$
$$
B \log (1 + x) (1 + \log (1 + x)) \leq x.
$$
$$
B \log (1 + x) (1 + \log (1 + x)) \leq B (1 + \log (1 + x)) ^ {2} \leq B (1 + \log (2 x)) ^ {2} \leq B [ \log (6 x) ] ^ {2}.
$$
We define  $f(x) \coloneqq x - B[\log (6x)]^2$ , then we have  $f'(x) = 1 - \frac{2B\log(6x)}{x}$ . Since  $x \geq 2C_1B\log (B)$ , we have
$$
f ^ {\prime} (x) \geq 1 - \frac {\log \left(1 2 C _ {1} B \log (B)\right)}{C _ {1} \log (B)}.
$$
We can take  $C_1 \geq 1$  such that  $C_1 \log(B) - \log(12C_1 B \log(B)) \geq 0$  whenever  $B \geq 16$ . Therefore, we know  $f(x)$  is increasing when  $x \geq C_1 B \log(B)^2$ . Then, it suffices to prove
$$
\left[ \log \left(6 C _ {1} B \log (B) ^ {2}\right) \right] ^ {2} \leq C _ {1} \log (B) ^ {2}.
$$
Since  $\left[\log \left(6C_1B\log (B)^2\right)\right]^2\leq 2\log (B)^2 +2\left[\log \left(6C_1\log (B)^2\right)\right]^2$  , it suffices to prove
$$
\log \left(6 C _ {1} \log (B) ^ {2}\right) \leq \sqrt {\frac {C _ {1} - 2}{2}} \log (B).
$$
When  $C_1 \geq 4$ , the difference of right hand side and left hand side is always increasing w.r.t.  $B$  for fixed  $C_1$ . Therefore, it suffices to prove the case when  $B = 16$ . Noticing that we can always take a sufficiently large uniform constant  $C_1$  such that the inequality above holds when  $B = 16$ , we conclude.
Lemma D.9 (Chain rule of Kullback-Leibler divergence, Exercise 3.2 in [Wainwright, 2019]). Given two  $n$ -variate distributions  $\mathbb{Q}$  and  $\mathbb{P}$ , show that the Kullback-Leibler divergence can be decomposed as
$$
D (\mathbb {Q}; \mathbb {P}) = D \left(\mathbb {Q} _ {1}; \mathbb {P} _ {1}\right) + \sum_ {j = 2} ^ {n} \mathbb {E} _ {\mathbb {Q} _ {1} ^ {j - 1}} \left[ D \left(\mathbb {Q} _ {j} \left(\cdot \mid X _ {1} ^ {j - 1}\right); \mathbb {P} _ {j} \left(\cdot \mid X _ {1} ^ {j - 1}\right)\right) \right],
$$
where  $\mathbb{Q}_j\left(\cdot \mid X_1^{j - 1}\right)$  denotes the conditional distribution of  $X_{j}$  given  $(X_{1},\ldots ,X_{j - 1})$  under  $\mathbb{Q}$ , with a similar definition for  $\mathbb{P}_j\left(\cdot \mid X_1^{j - 1}\right)$ .
Lemma D.10 (Bretagnolle-Huber Inequality, Theorem 14.1 in [Lattimore and Szepesvári, 2020]). Let  $\mathbb{P}$  and  $\mathbb{Q}$  be probability measures on the same measurable space  $(\Omega, \mathcal{F})$ , and let  $A \in \mathcal{F}$  be an arbitrary event. Then,
$$
\mathbb {P} (A) + \mathbb {Q} \left(A ^ {c}\right) \geq \frac {1}{2} \exp (- \mathrm {D} (\mathbb {P}; \mathbb {Q}))
$$
where  $A^c = \Omega \backslash A$  is the complement of  $A$ .
Lemma D.11 (Value Difference Lemma, Lemma E.15 in [Dann et al., 2017]). For any two MDPs  $\mathcal{M}'$  and  $\mathcal{M}''$  with rewards  $r'$  and  $r''$  and transition probabilities  $\mathbb{P}'$  and  $\mathbb{P}''$ , the difference in values with respect to the same policy  $\pi$  can be written as
$$
\begin{array}{l} V _ {i} ^ {\prime} (s) - V _ {i} ^ {\prime \prime} (s) = \mathbb {E} ^ {\prime \prime} \left[ \sum_ {t = i} ^ {H} \left(r ^ {\prime} \left(s _ {t}, a _ {t}, t\right) - r ^ {\prime \prime} \left(s _ {t}, a _ {t}, t\right)\right) \mid s _ {i} = s \right] \\ + \mathbb {E} ^ {\prime \prime} \left[ \sum_ {t = i} ^ {H} \left(\mathbb {P} ^ {\prime} (s _ {t}, a _ {t}, t) - \mathbb {P} ^ {\prime \prime} (s _ {t}, a _ {t}, t)\right) ^ {\top} V _ {t + 1} ^ {\prime} \mid s _ {i} = s \right] \\ \end{array}
$$
where  $V_{H + 1}^{\prime}(s) = V_{H + 1}^{\prime \prime}(s) = 0$  for any state  $s$  and the expectation  $\mathbb{E}'$  is taken with respect to  $\mathbb{P}'$  and  $\pi$  and  $\mathbb{E}''$  with respect to  $\mathbb{P}''$  and  $\pi$ .
E Details of the planning phase
In this section, we provide some details of the planning phase in algorithm 3. In the planning phase, we are given a reward function  $r: \mathcal{S} \times \mathcal{A} \to [0,1]$  and we compute an estimate of sparsified transition dynamics  $\widetilde{\mathbb{P}}^{\dagger}$ , which is formally defined in appendix A.1.1. The goal of the planning phase is to compute the optimal policy  $\pi_{final}$  on the MDP specified by the transition dynamics  $\widetilde{\mathbb{P}}^{\dagger}$  and reward function  $r^{\dagger}$ , where  $r^{\dagger}$  is the sparsified version of  $r: r^{\dagger}(s,a) = r(s,a)$  and  $r^{\dagger}(s^{\dagger},a) = 0$  for any  $a \in \mathcal{A}$ . To compute the optimal policy, we iteratively apply the Bellman optimality equation. First, we define  $\widetilde{Q}_H(s,a) = r^{\dagger}(s,a)$  for any  $(s,a)$  and solve
$$
\pi_{final,H}(s) = \operatorname *{arg  max}_{a\in \mathcal{A}}r^{\dagger}(s,a).
$$
Then, for  $h = H - 1, H - 2, \ldots, 2, 1$ , we iteratively define
$$
\widetilde {Q} _ {h} (s, a) := r ^ {\dagger} (s, a) + \sum_ {s ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \widetilde {Q} _ {h + 1} \left(s ^ {\prime}, \pi_ {f i n a l, h + 1} \left(s ^ {\prime}\right)\right)
$$
for any  $(s,a)$ , and then solve
$$
\pi_{final,h}(s) = \operatorname *{arg  max}_{a\in \mathcal{A}}\widetilde{Q}_{h}(s,a)
$$
for any  $s \in S$ . For  $s^{\dagger}$  and any  $h \in [H]$ ,  $\pi_{final,h}(s^{\dagger})$  can be arbitrary action. Then, from the property of Bellman optimality equation, we know  $\pi_{final}$  is the optimal policy on  $\widetilde{\mathbb{P}}^{\dagger}$  and  $r^{\dagger}$ .
F More related works
Other low-switching algorithms Low-switching learning algorithms were initially studied in the context of bandits, with the UCB2 algorithm [Auer et al., 2002] achieving an  $O(\mathcal{A}\log K)$  switching
cost. Gao et al. [2019] demonstrated a sufficient and necessary  $O(\mathcal{A}\log \log K)$  switching cost for attaining the minimax optimal regret in multi-armed bandits. In both adversarial and stochastic online learning, [Cesa-Bianchi et al., 2013] designed an algorithm that achieves an  $O(\log \log K)$  switching cost.
Reward-free reinforcement learning In reward-free reinforcement learning (RFRL) the goal is to find a near-optimal policy for any given reward function. [Jin et al., 2020a] proposed an algorithm based on EULER [Zanette and Brunskill, 2019] that can find a  $\varepsilon$  policy with  $\widetilde{O}(H^5 |\mathcal{S}|^2 |\mathcal{A}| / \varepsilon^2)$  trajectories. Subsequently, [Kaufmann et al., 2021] reduces the sample complexity by a factor  $H$  by using uncertainty functions to upper bound the value estimation error. The sample complexity was further improved by another  $H$  factor by [Ménard et al., 2021].
A lower bound of  $\widetilde{O}(H^2 |\mathcal{S}|^2 |\mathcal{A}| / \varepsilon^2)$  was established for homogeneous MDPs by [Jin et al., 2020a], while an additional  $H$  factor is conjectured for non-homogeneous cases. [Zhang et al., 2021] proposed the first algorithm with matching sample complexity in the homogeneous case. Similar results are available with linear [Wang et al., 2020a, Wagenmaker et al., 2022, Zanette et al., 2020] and general function approximation [Chen et al., 2022, Qiu et al., 2021].
Offline reinforcement learning In offline reinforcement learning the goal is to learn a near-optimal policy from an existing dataset which is generated from a (possibly very different) logging policy. Offline RL in tabular domains has been investigated extensively [Yin and Wang, 2020, Jin et al., 2020c, Nachum et al., 2019, Rashidinejad et al., 2021, Kallus and Uehara, 2022, Xie and Jiang, 2020a]. Similar results were shown using linear [Yin et al., Xiong et al., 2022, Nguyen-Tang et al., 2022, Zanette et al., 2021b] and general function approximation[Xie et al., 2021a, Long et al., 2021, Zhang et al., 2022, Duan et al., 2021, Jiang and Huang, 2020, Uehara and Sun, 2021, Zanette and Wainwright, 2022, Rashidinejad et al., 2022, Yin et al., 2022]. Offline RL is effective when the dataset 'covers' a near optimal policy, as measured by a certain concentrability factor. In the function approximation setting additional conditions, such as Bellman completeness, may need to be approximately satisfied [Munos and Szepesvári, 2008, Chen and Jiang, 2019, Zanette, 2023, Wang et al., 2020b, Foster et al., 2021, Zhan et al., 2022].
Task-agnostic reinforcement learning Another related line of work is task-agnostic RL, where  $N$  tasks are considered during the planning phase, and the reward functions is collected from the environment instead of being provided directly. [Zhang et al., 2020a] presented the first task-agnostic algorithm, UBEZero, with a sample complexity of  $\tilde{O}(H^5 |\mathcal{S}| |\mathcal{A}| \log(N) / \varepsilon^2)$ . Recently, [Li et al., 2023a] proposed an algorithm that leverages offline RL techniques to estimate a well-behaved behavior policy in the reward-agnostic phase, achieving minimax sample complexity. Other works exploring effective exploration schemes in RL include [Hazan et al., 2019, Du et al., 2019, Misra et al., 2020]. [Li et al., 2023b] also considered an offline-to-online reinforcement learning algorithm which explores the environment using two mixed policies in a reward-free mode.