SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In case  $x^{\star} = 0$ , we upper bound the minimum on the right hand side with the same expression evaluated at  $p := 1 / T$ . As a result,
$$
\begin{array}{l} \left[ \sum_ {t = 1} ^ {T} (\hat {c} ^ {t} + \hat {c} ^ {t - 1}) (y _ {1} ^ {t} - 0) \right] \leq \sum_ {t = 1} ^ {T} (\hat {c} ^ {t} + \hat {c} ^ {t - 1}) \frac {1}{T} - \frac {\log (\frac {1}{T}) + \log (1 - \frac {1}{T})}{\gamma} \\ \leq 2 + \frac {\log (T) + \log \left(\frac {T}{T - 1}\right)}{\gamma} \leq 2 + \frac {2 \log T}{\gamma} \tag {5} \\ \end{array}
$$
In case  $x^{\star} = 1$ , we upper bound the minimum on the right hand side by the expression evaluated at  $p := 1 - 1 / T$ . As a result,
$$
\begin{array}{l} \left[ \sum_ {t = 1} ^ {T} \left(\hat {c} ^ {t} + \hat {c} ^ {t - 1}\right) \left(y _ {1} ^ {t} - 1\right) \right] \leq \sum_ {t = 1} ^ {T} \left(\hat {c} ^ {t} + \hat {c} ^ {t - 1}\right) \left(1 - \frac {1}{T}\right) - \frac {\log \left(\frac {1}{T}\right) + \log \left(1 - \frac {1}{T}\right)}{\gamma} - \sum_ {t = 1} ^ {T} \left(\hat {c} ^ {t} + \hat {c} ^ {t - 1}\right) \\ \leq 2 + \frac {\log (T) + \log \left(\frac {T}{T - 1}\right)}{\gamma} \leq 2 + \frac {2 \log T}{\gamma} \tag {6} \\ \end{array}
$$
Therefore putting together Equation (5) and Equation (6), we can conclude that  $\sum_{t=1}^{T}(c^t + c^{t-1})^\top x^t - \min_{i\in[2]}\sum_{t=1}^{T}(c_i^t + c_i^{t-1}) \leq 2\log T/\gamma + 2$ .
A.3 Proof of Lemma 3.3
Before presenting the formal proof of Lemma 3.3 we present Lemma A.3 and Lemma A.4 that are necessary for its proof.
Lemma A.3. Let  $x^t$  as in Algorithm 2 and  $y_1^t$  be the BTRL update as in Lemma 3.2 with  $R(x) = -\log x - \log (1 - x)$  and  $x_1^t$  be the update as in Algorithm 3. Then the following hold.
-  $|x_1^t - y_1^t| \leq 8\gamma$
-  $|x_1^t - x_1^{t + 1}| \leq 16\gamma$
-  $\left| {y}_{1}^{t} - {y}_{1}^{t + 1}\right|  \leq  {8\gamma }$
Proof. Notice that for any  $x, y \in \Delta_2$  and cost vector  $c = (c_1, c_2) \in \mathbb{R}^2$ , we have that
$$
c ^ {\top} (x - y) = c _ {1} (x _ {1} - y _ {1}) + c _ {2} (x _ {2} - y _ {2}) = c _ {1} (x _ {1} - y _ {1}) + c _ {2} (- x _ {1} + y _ {1}) = (x _ {1} - y _ {1}) (c _ {1} - c _ {2}).
$$
This means that we can reduce the bidimensional update in Algorithm 2 as
$$
x _ {1} ^ {t} = \underset {p \in [ 0, 1 ]} {\arg \min } \left[ 2 \left(c _ {1} ^ {t - 1} - c _ {2} ^ {t - 1}\right) p + \sum_ {s = 1} ^ {t - 1} \left(c _ {1} ^ {s} - c _ {2} ^ {s} + c _ {1} ^ {s - 1} - c _ {2} ^ {s - 1}\right) p - \frac {\log p + \log (1 - p)}{\gamma} \right] \tag {7}
$$
At this point, using strong convexity of the log barrier function (Lemma A.1), the form of the updates in Equation (4) and Equation (7), we can invoke Lemma A.2 using  $x = x_{1}^{t}$  and  $y = y_{1}^{t}$ , this gives
$$
\left| x _ {1} ^ {t} - y _ {1} ^ {t} \right| \leq 2 \gamma \left| 2 c _ {1} ^ {t - 1} - 2 c _ {2} ^ {t - 1} - c _ {1} ^ {t} - c _ {1} ^ {t - 1} + c _ {2} ^ {t} + c _ {2} ^ {t - 1} \right| \leq 8 \gamma
$$
where we used that the cost sequence is in  $[-1, 1]$ . For the second fact, we invoke again Lemma A.2 but with  $x = x_1^t$  and  $y = x_1^{t + 1}$  and we obtain
$$
\left| x _ {1} ^ {t} - x _ {1} ^ {t + 1} \right| \leq 2 \gamma \left| 2 c _ {1} ^ {t - 1} - 2 c _ {2} ^ {t - 1} - c _ {1} ^ {t} - c _ {1} ^ {t - 1} + c _ {2} ^ {t} + c _ {2} ^ {t - 1} - 2 c _ {1} ^ {t} + 2 c _ {2} ^ {t} \right| \leq 1 6 \gamma
$$
For the third fact, we use Lemma A.2 but with  $x = y_1^t$  and  $y = y_1^{t + 1}$  and we obtain
$$
\left| y _ {1} ^ {t} - y _ {1} ^ {t + 1} \right| \leq 2 \gamma \left| c _ {1} ^ {t + 1} - c _ {2} ^ {t + 1} - c _ {1} ^ {t} + c _ {2} ^ {t} \right| \leq 8 \gamma
$$
Lemma A.4. Let  $(x,y) \in [0,1]^2$  and  $(x',y') \in [0,1]^2$  such that  $|x - y| \leq B$ ,  $|x - y'| \leq B$ ,  $|x' - y| \leq B$  and  $|x' - y'| \leq B$  with  $B \leq \frac{1}{8}$  then
$$
\left| A ^ {- 1} (x, y) - A ^ {- 1} \left(x ^ {\prime}, y ^ {\prime}\right) \right| \leq 1 9 2 \left| x - x ^ {\prime} \right| + 1 9 2 \left| y - y ^ {\prime} \right|
$$
where  $A(x,y) = (xy)^{-1} - (1 - x)^{-1}(1 - y)^{-1}$ .
Proof. To simplify notation we denote  $x_{t} \coloneqq tx + (1 - t)x'$  and  $y_{t} \coloneqq ty + (1 - t)y'$ . Then
$$
\begin{array}{l} A ^ {- 1} (x, y) - A ^ {- 1} \left(x ^ {\prime}, y ^ {\prime}\right) = \int_ {0} ^ {1} \left\langle \nabla A ^ {- 1} \left(x _ {t}, y _ {t}\right), \left(x, y\right) - \left(x ^ {\prime}, y ^ {\prime}\right) \right\rangle \partial t \\ \leq \max  _ {t \in [ 0, 1 ]} \| \nabla A ^ {- 1} \left(x _ {t}, y _ {t}\right) \| _ {\infty} \cdot \| \left(x, y\right) - \left(x ^ {\prime}, y ^ {\prime}\right) \| _ {1} \tag {8} \\ \end{array}
$$
Let us focus on bounding  $\| \nabla A^{-1}(x_t,y_t)\|_{\infty}$ . Notice that
$$
\left| \frac {\partial A ^ {- 1} \left(x _ {t} , y _ {t}\right)}{\partial x} \right| \leq \frac {3}{\left((1 - x _ {t}) \left(1 - y _ {t}\right) + x _ {t} y _ {t}\right) ^ {2}}. \tag {9}
$$
Now, notice that  $|x_{t} - y_{t}| \leq t|x - y| + (1 - t)|x' - y'| \leq B$ . Using the latter we can lower bound the denominator of Equation 9. More precisely,
$$
\begin{array}{l} (1 - x _ {t}) (1 - y _ {t}) + x _ {t} y _ {t} = x _ {t} ^ {2} + (1 - x _ {t}) ^ {2} + (2 x _ {t} - 1) (y _ {t} - x _ {t}) \\ \geq \frac {1}{4} - | 2 x _ {t} - 1 | | y _ {t} - x _ {t} | \\ \geq \frac {1}{4} - B \\ \end{array}
$$
So for  $B \leq \frac{1}{8}$  we obtain
$$
\left| \frac {\partial A ^ {- 1} (x _ {t} , y _ {t})}{\partial x} \right| \leq 3 \cdot 8 ^ {2} = 1 9 2.
$$
By symmetry, we can bound with analogous steps the partial derivative wrt to  $y$  and hence we get
$$
\| \nabla A ^ {- 1} (x _ {t}, y _ {t}) \| _ {\infty} \leq 1 9 2.
$$
Plugging this bound back in Equation (8) concludes the proof.
Proof. In order to prove this Lemma 3.3, we use the equivalent one-dimensional description provided in Equation 10.
$$
x _ {1} ^ {t} = \underset {p \in [ 0, 1 ]} {\arg \min } \left[ 2 \left(c _ {1} ^ {t - 1} - c _ {2} ^ {t - 1}\right) p + \sum_ {s = 1} ^ {t - 1} \left(c _ {1} ^ {s} - c _ {2} ^ {s} + c _ {1} ^ {s - 1} - c _ {2} ^ {s - 1}\right) p - \frac {\log p + \log (1 - p)}{\gamma} \right]. \tag {10}
$$
Similarly the update of BTRL in Lemma 3.2 can be equivalently described as,
$$
y _ {1} ^ {t} = \underset {p \in [ 0, 1 ]} {\arg \min } \left[ \sum_ {s = 1} ^ {t} \left(c _ {1} ^ {s} - c _ {2} ^ {s} + c _ {1} ^ {s - 1} - c _ {2} ^ {s - 1}\right) p - \frac {\log p + \log (1 - p)}{\gamma} \right]. \tag {11}
$$
Since  $\lim_{p\to \partial [0,1]}R(p) = \infty$  both  $x_1^t,y_1^t\in [0,1]\setminus \partial [0,1]$ . Therefore, the first order optimality for Equation (7) requires that
$$
2 \gamma \left(c _ {1} ^ {t - 1} - c _ {2} ^ {t - 1}\right) + \gamma \sum_ {s = 1} ^ {t - 1} c _ {1} ^ {s} + c _ {1} ^ {s - 1} - \left(c _ {2} ^ {s} + c _ {2} ^ {s - 1}\right) - \frac {1}{x _ {1} ^ {t}} + \frac {1}{1 - x _ {1} ^ {t}} = 0 \tag {12}
$$
Using the same reasoning for the BTRL updates in Equation (4)
$$
\gamma \left(c _ {1} ^ {t} - c _ {2} ^ {t}\right) + \gamma \left(c _ {1} ^ {t - 1} - c _ {2} ^ {t - 1}\right) + \gamma \sum_ {s = 1} ^ {t - 1} \left(c _ {1} ^ {s} + c _ {1} ^ {s - 1}\right) - \left(c _ {2} ^ {s} + c _ {2} ^ {s - 1}\right) - \frac {1}{y _ {1} ^ {t}} + \frac {1}{1 - y _ {1} ^ {t}} = 0. \tag {13}
$$
Now, subtracting Equation (12) to Equation (13), we obtain
$$
\gamma (c _ {1} ^ {t} - c _ {2} ^ {t} - c _ {1} ^ {t - 1} + c _ {2} ^ {t - 1}) - \frac {1}{y _ {1} ^ {t}} + \frac {1}{x _ {1} ^ {t}} + \frac {1}{1 - y _ {1} ^ {t}} - \frac {1}{1 - x _ {1} ^ {t}} = 0
$$
that implies
$$
\gamma (c _ {1} ^ {t} - c _ {2} ^ {t}) - \gamma (c _ {1} ^ {t - 1} - c _ {2} ^ {t - 1}) = (x _ {1} ^ {t} - y _ {1} ^ {t}) \underbrace {\left(\frac {1}{x _ {1} ^ {t} y _ {1} ^ {t}} + \frac {1}{(1 - y _ {1} ^ {t}) (1 - x _ {1} ^ {t})}\right)} _ {A (x _ {1} ^ {t}, y _ {1} ^ {t})}.
$$
Therefore, we can express the difference between the updates as a function of the costs according to the following formula
$$
x _ {1} ^ {t} - y _ {1} ^ {t} = \gamma A ^ {- 1} \left(x _ {1} ^ {t}, y _ {1} ^ {t}\right) \left(\left(c _ {1} ^ {t} - c _ {2} ^ {t}\right) - \left(c _ {1} ^ {t - 1} - c _ {2} ^ {t - 1}\right)\right). \tag {14}
$$
We conclude the proof by establishing that
$$
\left| A ^ {- 1} \left(x _ {1} ^ {t}, y _ {1} ^ {t}\right) - A ^ {- 1} \left(x _ {1} ^ {t + 1}, y _ {1} ^ {t + 1}\right) \right| \leq \mathcal {O} (\gamma).
$$
By Lemma A.3 we are ensured that
In case  $\gamma \leq 1 / (16\cdot 8)$  we are ensured that the conditions of Lemma A.4 are satisfied ( $B\leq 1 / 8$ ) and thus
$$
\left| A ^ {- 1} \left(x _ {1} ^ {t}, y _ {1} ^ {t}\right) - A ^ {- 1} \left(x _ {1} ^ {t + 1}, y _ {1} ^ {t + 1}\right) \right| \leq 1 9 2 \left(\left| x _ {1} ^ {t} - x _ {1} ^ {t + 1} \right| + \left| y _ {1} ^ {t} - y _ {1} ^ {t + 1} \right|\right)
$$
Combining the latter with the guarantees of Lemma A.4 we get that
$$
\left| A ^ {- 1} \left(x _ {1} ^ {t}, y _ {1} ^ {t}\right) - A ^ {- 1} \left(x _ {1} ^ {t + 1}, y _ {1} ^ {t + 1}\right) \right| \leq 1 9 2 \cdot 2 4 \gamma
$$
B Omitted proofs for the  $n$  dimensional case.
B.1 Auxiliary Lemmas
Corollary B.1. i)  $Q(s,\pi ,c) = q(s,\pi ,c)^{\top}\cdot \pi (s)ii)c^{\top}x^{\pi} = Q(\mathrm{root},\pi ,c).$
Proof. For fact i) for any  $s \in \operatorname{Level}(h)$ , we have that
$$
\begin{array}{l} Q (s, \pi , c) = \sum_ {i \in L} \Pr (s, i, \pi) c _ {i} \\ = \sum_ {i \in L} \pi (\ell (s) | s) \Pr (\ell (s), i, \pi) c _ {i} + \sum_ {i \in L} \pi (r (s) | s) \Pr (r (s), i, \pi) c _ {i} \\ = \pi (\ell (s) | s) \sum_ {h \in L} \Pr (\ell (s), i, \pi) c _ {i} + \pi (r (s) | s) \sum_ {i \in L} \Pr (r (s), i, \pi) c _ {i} \\ = \pi (\ell (s) | s) Q (\ell (s), \pi , c) + \pi (r (s) | s) Q (r (s), \pi , c) \\ = q (s, \pi , c) ^ {\top} \cdot \pi (s) \\ \end{array}
$$
where the second last equality uses the fact that  $s \in \operatorname{Level}(h) \implies \ell(s), r(s) \in \operatorname{Level}(h + 1)$ .
Finally, fact ii) follows trivially from the definition of  $x^{\pi}$ . Indeed, we have that
$$
c ^ {\top} \cdot x ^ {\pi} = \sum_ {i \in L} x ^ {\pi} (i) c _ {i} = \sum_ {i \in L} \Pr (\operatorname {r o o t}, i, \pi) c _ {i} = Q (\operatorname {r o o t}, \pi , c)
$$
B.2 Proof of Lemma 3.8
Lemma 3.8. Let a leaf node  $i \in L$  and let  $p = (\mathrm{root} = s_1, \dots, s_H = i)$  denotes the path from the root to  $i$ . Then the following holds,
$$
\sum_ {t = 1} ^ {T} \left(c ^ {t} + c ^ {t - 1}\right) ^ {\top} \cdot x ^ {\pi^ {t}} - 2 \sum_ {t = 1} ^ {T} c _ {i} ^ {t} \leq \sum_ {s _ {\ell} \in p} \mathcal {R} _ {\mathrm {l o c}} (s _ {\ell})
$$
Proof. By Item 2 of Corollary B.1 and the fact that  $c^0 = 0$ , we get
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(c ^ {t} + c ^ {t - 1}\right) ^ {\top} \cdot x ^ {\pi^ {t}} - 2 \sum_ {t = 1} ^ {T} c _ {i} ^ {t} = \sum_ {t = 1} ^ {T} \left(Q (\operatorname {r o o t}, \pi^ {t}, c ^ {t}) + Q (\operatorname {r o o t}, \pi^ {t}, c ^ {t - 1}) - Q (i, \pi^ {t}, c ^ {t}) - Q (i, \pi^ {t}, c ^ {t - 1})\right) \\ = \sum_ {t = 1} ^ {T} \sum_ {\ell = 1} ^ {H - 1} \left(Q \left(s _ {\ell}, \pi^ {t}, c ^ {t}\right) + Q \left(s _ {\ell}, \pi^ {t}, c ^ {t - 1}\right) - Q \left(s _ {\ell + 1}, \pi^ {t}, c ^ {t}\right) - Q \left(s _ {\ell + 1}, \pi^ {t}, c ^ {t - 1}\right)\right) \\ = \sum_ {t = 1} ^ {T} \sum_ {\ell = 1} ^ {H - 1} \left(Q \left(s _ {\ell}, \pi^ {t}, c ^ {t}\right) + Q \left(s _ {\ell}, \pi^ {t}, c ^ {t - 1}\right)\right)) \\ - \min  _ {\alpha \in \left\{\ell \left(s _ {\ell}\right), r \left(s _ {\ell}\right) \right\}} \sum_ {t = 1} ^ {T} \left(Q \left(\alpha , \pi^ {t}, c ^ {t}\right) + Q \left(\alpha , \pi^ {t}, c ^ {t - 1}\right)\right) \\ = \sum_ {t = 1} ^ {T} \sum_ {\ell = 1} ^ {H - 1} \left(q \left(s _ {\ell}, \pi^ {t}, c ^ {t}\right) + q \left(s _ {\ell}, \pi^ {t}, c ^ {t - 1}\right)\right) ^ {\top} \cdot \pi^ {t} \left(s _ {\ell}\right) \\ - \min  _ {\alpha \in \left\{\ell \left(s _ {\ell}\right), r \left(s _ {\ell}\right) \right\}} \sum_ {t = 1} ^ {T} \left(Q \left(\alpha , \pi^ {t}, c ^ {t}\right) + Q \left(\alpha , \pi^ {t}, c ^ {t - 1}\right)\right) \quad \text {C o r o l l a r y B . 1} \\ = \sum_ {s _ {\ell} \in p} \mathcal {R} _ {\mathrm {l o c}} (s _ {\ell}) \\ \end{array}
$$
B.3 Proof of Lemma 3.9
Lemma 3.9. Let  $\pi^1, \ldots, \pi^T$  the policies produced by Algorithm 3 then for any state  $s \in V$ ,  $i) \| \pi^t(s) - \pi^{t-1}(s) \|_1 \leq 48\gamma$  and  $ii) \| q(s, \pi^t, c^{t-1}) - q(s, \pi^{t-1}, c^{t-1}) \|_\infty \leq 48\gamma \log n$ .
Proof. We first establish that  $\| \pi^t (s) - \pi^{t - 1}(s)\| _1\leq 48\gamma$
Let  $\bar{Q}(s, \pi, c) \coloneqq Q(\ell(s), \pi, c) - Q(r(s), \pi, c)$  then policy update in Step 6 of Algorithm 3 admits the following one-dimensional form,
$$
\pi^ {t} (\ell (s) | s) = \underset {x \in [ 0, 1 ]} {\arg \min } \left[ 2 \gamma \left(\bar {Q} \left(s, \pi^ {t}, c ^ {t}\right) + \bar {Q} \left(s, \pi^ {t}, c ^ {t - 1}\right)\right) + \gamma \sum_ {\tau = 1} ^ {t - 1} \left(\bar {Q} \left(s, \pi^ {\tau}, c ^ {\tau}\right) + \bar {Q} \left(s, \pi^ {\tau}, c ^ {\tau - 1}\right)\right) + R (x) \right].
$$
Similarly for the policy  $\pi^{t - 1}$
$$
\pi^ {t - 1} (\ell (s) | s) = \underset {x \in [ 0, 1 ]} {\arg \min } \left[ 2 \gamma (\bar {Q} (s, \pi^ {t - 1}, c ^ {t - 1}) + \bar {Q} (s, \pi^ {t - 1}, c ^ {t - 2})) + \gamma \sum_ {\tau = 1} ^ {t - 2} (\bar {Q} (s, \pi^ {\tau}, c ^ {\tau}) + \bar {Q} (s, \pi^ {\tau}, c ^ {\tau - 1})) + R (x) \right].
$$
Using Lemma A.2 we get that,
$$
\begin{array}{l} \left| \pi^ {t} (\ell (s) | s) - \pi^ {t - 1} (\ell (s) | s) \right| \leq 2 \gamma \left| 2 \bar {Q} (s, \pi^ {t}, c ^ {t}) + 2 \bar {Q} (s, \pi^ {t}, c ^ {t - 1}) + \bar {Q} (s, \pi^ {t - 1}, c ^ {t - 1}) + \bar {Q} (s, \pi^ {t - 1}, c ^ {t - 2}) \right. \\ - 2 \bar {Q} (s, \pi^ {t - 1}, c ^ {t - 1}) - 2 \bar {Q} (s, \pi^ {t - 1}, c ^ {t - 2}) \Bigg | \leq 2 4 \gamma \\ \end{array}
$$
where the last inequality comes from the fact that  $-1\leq Q(s,\pi ,c)\leq 1$  and thus  $|\bar{Q} (s,\pi ,c)|\leq 2$
Up next we establish that
$$
\left\| q \left(s, \pi^ {t}, c ^ {t - 1}\right) - q \left(s, \pi^ {t - 1}, c ^ {t - 1}\right) \right\| _ {\infty} \leq 4 8 \gamma \log n.
$$
To simplify notation we let  $h_0$  denote the depth of state  $s_0 \in V$ . Up next we prove that
$$
\left\| q \left(s _ {0}, \pi^ {t}, c ^ {t - 1}\right) - q \left(s _ {0}, \pi^ {t - 1}, c ^ {t - 1}\right) \right\| _ {\infty} \leq 4 8 \gamma \log n.
$$
In order to prove the latter we deploy a coupling argument by considering two correlated random walks  $(s_0, s_0), (s_1, s_1'), \ldots, (s_H, s_H')$  where both walks are initialized at  $(s_0, s_0)$  while at each level  $h \in \{h_0, \ldots, H - 1\}$ , the first walk marginally follows policy  $\pi \in \Delta_2$  while the second walk marginally follows  $\pi' \in \Delta_2$ .
More precisely, let  $(s_h,s_h^{\prime})$  the pair of nodes visited respectively by the first and the second walk at level  $h\in \{h_0,\dots ,H - 1\}$ . Then the next pair of nodes  $(s_h,s_h^{\prime})$  follows the following joint probability distribution.
- In case  $s_h' \neq s_h$ : The next pair of nodes  $(s_{h+1}, s_{h+1}')$  are independent random variables respectively following  $\pi(s_h) \in \Delta_2$  and  $\pi'(s_h') \in \Delta_2$ . More precisely,
$$
s _ {h + 1} = \left\{ \begin{array}{r l} & {\ell (s _ {h}) \quad \mathrm {w . p .} \quad \pi (\ell (s _ {h}) | s _ {h})} \\ & {r (s _ {h}) \quad \mathrm {w . p .} \quad 1 - \pi (\ell (s _ {h}) | s _ {h})} \end{array} \right. \mathrm {a n d} s _ {h + 1} ^ {\prime} = \left\{ \begin{array}{r l} & {\ell (s _ {h} ^ {\prime}) \quad \mathrm {w . p .} \quad \pi^ {\prime} (\ell (s _ {h} ^ {\prime}) | s _ {h} ^ {\prime})} \\ & {r (s _ {h} ^ {\prime}) \quad \mathrm {w . p .} \quad 1 - \pi^ {\prime} (\ell (s _ {h} ^ {\prime}) | s _ {h} ^ {\prime})} \end{array} \right.
$$
- In case  $s_h = s_h' = s$  and  $\pi(\ell(s)|s) \leq \pi'(\ell(s)|s)$ : Then the next pair of nodes  $(s_j, s_h')$  follows the joint probability distribution,
$$
(s _ {h + 1}, s _ {h + 1} ^ {\prime}) = \left\{ \begin{array}{l l} & (\ell (s), \ell (s)) \quad \text {w . p .} \quad \pi (\ell (s) | s) \\ & (r (s), \ell (s)) \quad \text {w . p .} \quad \pi^ {\prime} (\ell (s) | s) - \pi (\ell (s) | s) \\ & (r (s), r (s)) \quad \text {w . p .} \quad 1 - \pi^ {\prime} (\ell (s) | s) \end{array} \right.
$$
- In case  $s_h = s_h' = s$  and  $\pi(\ell(s)|s) \geq \pi'(\ell(s)|s)$ : Then the next pair of nodes  $(s_j, s_h')$  follows the joint probability distribution,
$$
(s _ {h + 1}, s _ {h + 1} ^ {\prime}) = \left\{ \begin{array}{l l} & (\ell (s), \ell (s)) \quad \text {w . p .} \quad \pi^ {\prime} (\ell (s) | s) \\ & (\ell (s), r (s)) \quad \text {w . p .} \quad \pi (\ell (s) | s) - \pi^ {\prime} (\ell (s) | s) \\ & (r (s), r (s)) \quad \text {w . p .} \quad 1 - \pi (\ell (s) | s) \end{array} \right.
$$
The above joint random walk, guarantees that the first random walk (resp. the second) follows policy  $\pi$  (resp.  $\pi^{\prime}$  for the second coordinate). More precisely,
$$
\Pr \left[ s _ {h + 1} = \ell (s) \mid s _ {h} = s \right] = \pi (\ell (s) | s) \text {a n d} \Pr \left[ s _ {h + 1} ^ {\prime} = \ell (s) \mid s _ {h} ^ {\prime} = s \right] = \pi^ {\prime} (\ell (s) | s)
$$
$$
\operatorname {E} \left[ c _ {i} - c _ {i ^ {\prime}} \right] = Q (s _ {0}, \pi , c) - Q (s _ {0}, \pi^ {\prime}, c)
$$
where  $(i,i^{\prime})\in L\times L$  denotes the pair of leaves reached by the joint random walk initialized at  $s_0\in V / L$
$$
\begin{array}{l} \left| Q \left(s _ {0}, \pi , c\right) - Q \left(s _ {0}, \pi^ {\prime}, c\right) \right| = \left| \mathrm {E} \left[ c _ {i} - c _ {i ^ {\prime}} \right] \right| \leq \mathrm {E} \left[ \left| c _ {i} - c _ {i ^ {\prime}} \right| \right] \\ = \sum_ {h = h _ {0}} ^ {H - 1} \sum_ {s \in \operatorname {L e v e l} (h)} \operatorname {E} \left[ | c _ {i} - c _ {i ^ {\prime}} | \mid s _ {h + 1} ^ {\prime} \neq s _ {h + 1}, s _ {h} ^ {\prime} = s _ {h} = s \right] \operatorname {P} \left[ s _ {h + 1} ^ {\prime} \neq s _ {h + 1}, s _ {h} ^ {\prime} = s _ {h} = s \right] \\ \leq 2 \sum_ {h = h _ {0}} ^ {H - 1} \sum_ {s \in \operatorname {L e v e l} (h)} \mathrm {P} \left[ s _ {h + 1} ^ {\prime} \neq s _ {h + 1}, s _ {h} ^ {\prime} = s _ {h} = s \right] \\ = 2 \sum_ {h = h _ {0}} ^ {H - 1} \sum_ {s \in \operatorname {L e v e l} (h)} \mathrm {P} \left[ s _ {h + 1} ^ {\prime} \neq s _ {h + 1} | s _ {h} ^ {\prime} = s _ {h} = s \right] \mathrm {P} \left[ s _ {h} ^ {\prime} = s _ {h} = s \right] \\ \leq 2 \sum_ {h = h _ {0}} ^ {H - 1} \sum_ {s \in \operatorname {L e v e l} (h)} | \pi (\ell (s) | s) - \pi^ {\prime} (\ell (s) | s) | P \left[ s _ {h} ^ {\prime} = s _ {h} = s \right] \\ \end{array}
$$
where in the second equality we used the fact that  $\{s_{h + 1}^{\prime}\neq s_{h + 1},s_{h}^{\prime} = s_{h} = s\}_{s\in V,h\in [H]}$  are disjoint events.
By setting  $\pi' = \pi^t$  and  $\pi = \pi^{t-1}$  we get that
$$
\begin{array}{l} \left| Q (s _ {0}, \pi^ {t}, c) - Q (s _ {0}, \pi^ {t - 1}, c) \right| \leq 2 \sum_ {h = h _ {0}} ^ {H - 1} \sum_ {s \in \operatorname {L e v e l} (h)} \left| \pi^ {t} (\ell (s) | s) - \pi^ {t - 1} (\ell (s) | s) \right| \mathrm {P} \left[ s _ {h} ^ {\prime} = s _ {h} = s \right] \\ \leq 48 \gamma \sum_ {h = h _ {0}} ^ {H - 1} \sum_ {s \in \operatorname {L e v e l} (h)} \mathrm {P} \left[ s _ {h} ^ {\prime} = s _ {h} = s \right] \\ = 4 8 \gamma \sum_ {h = h _ {0}} ^ {H - 1} 1 = 4 8 \gamma \log n \\ \end{array}
$$
$$
\| q (s _ {0}, \pi^ {t}, c ^ {t - 1}) - q (s _ {0}, \pi^ {t - 1}, c ^ {t - 1}) \| _ {\infty} = \max _ {\alpha \in \{\ell (s _ {0}), r (s _ {0}) \}} \left| Q (\alpha , \pi^ {t}, c ^ {t - 1}) - Q (\alpha , \pi^ {t - 1}, c ^ {t - 1}) \right| \leq 4 8 \gamma \log n.
$$
B.4 Proof of Lemma 3.10
Proof. Let the step-size  $\gamma > 0$  of Algorithm 3 defined as  $\gamma := \frac{1}{32 \cdot 8} \left( \frac{\log(T)}{T \log n} \right)^{\frac{1}{3}}$ . Let us also introduce the BTRL update for state  $s$  that is
$$
\tilde {\pi} ^ {t} (s) := \underset {x \in \Delta_ {2}} {\arg \min } \left[ \sum_ {\tau = 1} ^ {t} \left(q \left(s, \pi^ {\tau}, c ^ {\tau - 1}\right) + q \left(s, \pi^ {\tau}, c ^ {\tau}\right)\right) ^ {\top} x + R (x) / \gamma \right] \tag {15}
$$
We can bound two separate sources of regret, according to the decomposition
$$
\begin{array}{l} \mathcal {R} _ {l o c} ^ {T} (s) = \underbrace {\sum_ {t = 1} ^ {T} \left(q (s , \pi^ {t} , c ^ {t}) + q (s , \pi^ {t - 1} , c ^ {t})\right) ^ {\top} \cdot \tilde {\pi} ^ {t} (s) - \min  _ {\alpha \in \{\ell (s) , r (s) \}} \sum_ {t = 1} ^ {T} \left(Q (\alpha , \pi^ {t} , c ^ {t}) + Q (\alpha , \pi^ {t - 1} , c ^ {t})\right)} _ {\text {T e r m I}} \\ + \underbrace {\sum_ {t = 1} ^ {T} \left(q \left(s , \pi^ {t} , c ^ {t}\right) + q \left(s , \pi^ {t - 1} , c ^ {t}\right)\right) ^ {\top} \cdot \left(\pi^ {t} (s) - \tilde {\pi} ^ {t} (s)\right)} _ {\text {T e r m I I}} \tag {16} \\ \end{array}
$$
First, we recognize that Term I is the BTRL local regret, therefore applying Lemma 3.2, we have
$$
\text {T e r m} \mathrm {I} \leq \mathcal {O} \left(\frac {\log T}{\gamma}\right)
$$
Then, it remains to bound the term that quantifies the closeness between  $\pi^t$  and  $\tilde{\pi}^t$ , that is
$$
\sum_ {t = 1} ^ {T} \left(q (s, \pi^ {t}, c ^ {t}) + q (s, \pi^ {t - 1}, c ^ {t})\right) ^ {\top} \cdot \left(\pi^ {t} (s) - \tilde {\pi} ^ {t} (s)\right)
$$
Let  $\bar{Q}(s, \pi, c) \coloneqq Q(\ell(s), \pi, c) - Q(r(s), \pi, c)$  then by using Corollary B.1 we get that
$$
\sum_ {t = 1} ^ {T} \left(q (s, \pi^ {t}, c ^ {t}) + q (s, \pi^ {t - 1}, c ^ {t})\right) ^ {\top} \cdot \left(\pi^ {t} (s) - \tilde {\pi} ^ {t} (s)\right) = \sum_ {t = 1} ^ {T} \left[ \bar {Q} (s, \pi^ {t}, c ^ {t}) + \bar {Q} (s, \pi^ {t}, c ^ {t - 1}) \right] \cdot \left[ \pi^ {t} (\ell (s) | s) - \tilde {\pi} ^ {t} (\ell (s) | s) \right] \tag {17}
$$
At the same time by Lemma 3.3 we get that
$$
\pi^ {t} (\ell (s) | s) - \tilde {\pi} ^ {t} (\ell (s) | s) = \gamma \frac {\bar {Q} (s , \pi^ {t} , c ^ {t}) - \bar {Q} (s , \pi^ {t} , c ^ {t - 1})}{A \left(\pi^ {t} (\ell (s) | s), \tilde {\pi} ^ {t} (\ell (s) | s)\right)} \tag {18}
$$
Hence combining Equation 17 with Equation 18 we obtain
$$
\sum_ {t = 1} ^ {T} \left(q (s, \pi^ {t}, c ^ {t}) + q (s, \pi^ {t - 1}, c ^ {t})\right) ^ {\top} \cdot \left(\pi^ {t} (s) - \tilde {\pi} ^ {t} (s)\right) = \gamma \sum_ {t = 1} ^ {T} \frac {\bar {Q} ^ {2} (s , \pi^ {t} , c ^ {t}) - \bar {Q} ^ {2} (s , \pi^ {t} , c ^ {t - 1})}{A (\pi^ {t} (\ell (s) | s), \tilde {\pi} ^ {t} (\ell (s) | s))}
$$
At this point, we notice that unfortunately we can not rearrange the sum easily because of the term  $\bar{Q}^2 (s,\pi^t,c^{t - 1})$  that depends on both indices  $t$  and  $t - 1$ . To go around this issue, we add and subtract the term  $\frac{\bar{Q}^2(s,\pi^{t - 1},c^{t - 1})}{A(\pi^t(\ell(s)|s),\tilde{\pi}^t(\ell(s)|s))}$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(q (s, \pi^ {t}, c ^ {t}) + q (s, \pi^ {t - 1}, c ^ {t})\right) ^ {\top} \cdot \left(\pi^ {t} (s) - \tilde {\pi} ^ {t} (s)\right) = \gamma \sum_ {t = 1} ^ {T} \frac {\bar {Q} ^ {2} (s , \pi^ {t} , c ^ {t}) - \bar {Q} ^ {2} (s , \pi^ {t - 1} , c ^ {t - 1})}{A (\pi^ {t} (\ell (s) | s) , \tilde {\pi} ^ {t} (\ell (s) | s))} \\ + \gamma \sum_ {t = 1} ^ {T} \frac {\bar {Q} ^ {2} \left(s , \pi^ {t - 1} , c ^ {t - 1}\right) - \bar {Q} ^ {2} \left(s , \pi^ {t} , c ^ {t - 1}\right)}{A \left(\pi^ {t} (\ell (s) | s) , \tilde {\pi} ^ {t} (\ell (s) | s)\right)}. \tag {19} \\ \end{array}
$$
Now we bound the first term. Notice that the assumption of Lemma A.4 are satisfied with  $B = 32\gamma$  and that  $\gamma \leq (8 \cdot 32)^{-1}$  ensures  $B \leq \frac{1}{8}$ . Therefore, rearranging the sum and invoking Lemma A.4
for  $x = \pi^t$ ,  $y = \tilde{\pi}^t$ ,  $x' = \pi^{t+1}$ ,  $y' = \tilde{\pi}^{t+1}$ , we get