SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Focus on round  $t = A_{i+1}$ . Note that the history  $\mathrm{hist}_t$  contains exactly  $i$  agents that chose arm 1, both under event  $\mathrm{Fail}_1 \cap \mathrm{Fail}_2$  and under event  $\mathcal{E}$ . Yet, arm 2 is chosen under  $\mathcal{E}$ , while arm 1 is chosen under  $\mathrm{Fail}_1 \cap \mathrm{Fail}_2$ . This violates the monotonicity property (P2) from the theorem statement. Thus, we've proved (C.7) for any fixed realization of the types. Consequently, (C.7) holds in general.
C.4 Proof of Lemma C.1
Proof. We use the following sharp lower bound on the tail probability of binomial distribution.
Theorem C.9 (Theorem 9 in [61]). Let  $n \in \mathbb{N}$  be a positive integer and let  $(X_{i})_{i \in [n]}$  be a sequence of i.i.d Bernoulli random variables with prameter  $p$ . For any  $\beta > 1$  there exists constants  $c_{\beta}$  and  $C_{\beta}$  that only rely on  $\beta$ , such that for all  $x$  satisfying  $x \in [0, \frac{np}{\beta}]$  and  $x + n(1 - p) \geq 1$ , we have
$$
\Pr \left[ \sum_ {i = 1} ^ {n} X _ {i} \leq n p - x \right] \geq c _ {\beta} e ^ {- C _ {\beta} n D (p - \frac {x}{n} | | p)},
$$
where  $D(x||y)$  denotes the KL divergence between two Bernoulli random variables with parameters  $x$  and  $y$ .
We use the above result with  $x = n(p - q)$  and  $\beta = \frac{1 - c}{1 - \frac{9}{8}c}$ . Note that  $\beta > 1$  since  $c < \frac{1}{2}$ . We first verify that  $x, \beta$  satisfy the conditions of the lemma. The  $x + n(1 - p) \geq 1$  condition holds by the assumption  $n \geq 1 / c$ :
$$
x + n (1 - p) \geq n (1 - p) \geq n c \geq 1.
$$
As for the  $x \leq \frac{np}{\beta}$  condition, by definition of  $x$
$$
\frac {n p}{x} = \frac {n p}{n (p - q)} = \frac {p}{p - q}.
$$
Since  $p \leq 1 - c$  and  $\frac{p}{p - q}$  is decreasing in  $p$  for  $p \geq q$ , we can further bound this with
$$
\frac {p}{p - q} \geq \frac {1 - c}{1 - c - q} \geq \frac {1 - c}{1 - c - \frac {c}{8}} = \beta ,
$$
where the second inequality follows from  $q \geq c / 8$  and  $q < p \leq 1 - c$ , together with the fact that  $\frac{1 - c}{1 - c - q}$  is decreasing in  $q$  for  $q < 1 - c$ . We obtain  $x \leq \frac{np}{\beta}$  by rearranging.
Invoking Theorem C.9 with the given values, we obtain
$$
\Pr \left[ \frac {\sum_ {i = 1} ^ {n} X _ {i}}{n} \leq q \right] \geq c _ {\beta} e ^ {- C _ {\beta} n D (q \| p)} = \Omega \left(e ^ {- O (n D (q \| p))}\right). \tag {C.8}
$$
Next, we use the following type of reverse Pinsker's inequality to upper bound  $D(q||p)$ .
Theorem C.10 ([28]). For any two probability measures  $P$  and  $Q$  on a finite support  $X$ , if  $Q$  is absolutely continuous with respect to  $P$ , then the their KL divergence  $D(Q||P)$  is upper bounded by  $\frac{2}{\alpha_P} \delta(Q, P)^2$  where  $\alpha_P = \min_{x \in X} P(x)$  and  $\delta(Q, P)$  denotes the total variation distance between  $P$  and  $Q$ .
Setting  $P = \operatorname{Bernoulli}(p)$  and  $Q = \operatorname{Bernoulli}(q)$ , we have  $\alpha_{P} = \min(p, 1 - p)$ , and  $\delta(Q, P) = p - q$ . Therefore, since  $\min(p, 1 - p) \geq c$  by assumption, we conclude  $D(q||p) \leq O((p - q)^2)$ . Plugging this back in Equation (C.8) finishes the proof.
C.5 Proof of Lemma C.2
Our proof will rely on the following doob-style inequality for (super)martingales.
Lemma C.11 (Ville's Inequality [58]). Let  $(Z_{n})_{n\geq 0}$  be a positive supermartingale with respect to filtration  $(\mathcal{F}_n)_{n\geq 0}$ , i.e.  $Z_{n}\geq \mathbb{E}[Z_{n + 1}|\mathcal{F}_{n}]$  for any  $n\geq 0$ . Then the following holds for any  $x > 0$
$$
\Pr \left[ \max  _ {n \geq 0} Z _ {n} \geq x \right] \leq \mathbb {E} \left[ Z _ {0} \right] / x.
$$
In order to use this result, we will define the martingale  $Z_{n} \coloneqq u^{\sum_{i = 1}^{n}(X_{i + 1} - q)}$  for a suitable choice of  $u$  as specified by the following lemma.
Lemma C.12. Let  $c$  be an absolute constant. For any  $p \in (c, 1 - c)$  and  $q \in (0, p)$ , there exists a value of  $u \in (0, 1)$  such that
$$
(p \cdot u ^ {1 - q} + (1 - p) \cdot u ^ {- q}) = 1. \tag {C.9}
$$
In addition,  $u$  satisfies
$$
p \left(1 - u ^ {1 - q}\right) \geq \Omega (p - q). \tag {C.10}
$$
Proof. To see why such a  $u$  exists, define  $f(x) = (p \cdot x^{1 - q} + (1 - p) \cdot x^{-q})$ . It is clear that  $f(1) = 1$  and  $\lim_{x \to 0} f(x) = \infty$  as  $\lim_{x \to 0} (1 - p)x^{-q} = \infty$ . Furthermore,
$$
f ^ {\prime} (x) = p \cdot (1 - q) \cdot x ^ {- q} + (1 - p) \cdot (- q) \cdot x ^ {- q - 1},
$$
$$
f ^ {\prime} (1) = p (1 - q) - (1 - p) q = p - q > 0.
$$
Therefore,  $f(x)$  is decreasing at  $x = 1$ . Since  $\lim_{x\to 0}f(x) > f(1)$ , this implies that  $f(u) = f(1)$  for some  $u\in (0,1)$ , proving Equation (C.9).
We now prove Equation (C.10), define  $x_0$  as  $x_0 = \frac{(1 - p)q}{p(1 - q)}$ . Note that  $x_0 < 1$  since  $p > q$ . We claim that  $u \leq x_0$ . To see why, we first note that  $f'(x)$  can be rewritten as
$$
x ^ {- q - 1} (x p (1 - q) - (1 - p) q).
$$
It is clear that  $f'(x_0) = 0$ . Since  $xp(1 - q) - (1 - p)q$  is increasing in  $x$ , this further implies that  $f'(x) > 0$  for  $x > x_0$ . Now, if  $u > x_0$ , then since  $f'(x) > 0$  for  $x > x_0$ , we would conclude that  $f(u) < f(1)$ , which is not possible since  $f(u) = f(1) = 1$ . Therefore,  $u \leq x_0$  as claimed.
We now claim that  $x_0^{1 - q} \leq 1 - p + q$ . This would finish the proof since, together with  $u \leq x_0$ , this would imply
$$
p (1 - u ^ {1 - q}) \geq p (1 - x _ {0} ^ {1 - q}) \geq p (p - q) = \Omega (p - q),
$$
where for the last equation we have used the assumption  $p \in (c, 1 - c)$ .
To prove the claim, define  $\varepsilon \coloneqq p - q$ . We need to show that  $x_0^{1 - q} \leq 1 - \varepsilon$ , or equivalently  $\ln(x_0) \leq \frac{\ln(1 - \varepsilon)}{1 - q}$ . By definition of  $x_0$ , this is equivalent to
$$
\ln \left(\frac {(1 - p) (p - \varepsilon)}{(1 - p + \varepsilon) p}\right) \leq \frac {1}{1 - p + \varepsilon} \ln (1 - \varepsilon). \tag {C.11}
$$
Fix  $p$  and consider both hand sides as a function of  $\varepsilon$ . Putting  $\varepsilon = 0$ , both hands side coincide as they both equal 0. To prove Euqation (C.11), it suffices to show that as we increase  $\varepsilon$ , the left hand side decreases faster than the right hand side. Equivalently, we need to show that the derivative of the LHS with respect to  $\varepsilon$  is larger than the derivative of the RHS with respect to  $\varepsilon$  for  $\varepsilon \leq [0,p]$ . Taking the derivative with respect to  $\varepsilon$  on LHS, we obtain
$$
\frac {d}{d \varepsilon} \left(\ln (1 - p) + \ln (p - \varepsilon) - \ln (1 - p + \varepsilon) - \ln (p)\right) = - \frac {1}{p - \varepsilon} - \frac {1}{1 - p + \varepsilon}.
$$
Similarly taking the derivative on RHS we obtain
$$
\frac {d}{d \varepsilon} \left(\frac {\ln (1 - \varepsilon)}{1 - p + \varepsilon}\right) = - \frac {1}{(1 - \varepsilon) (1 - p + \varepsilon)} - \frac {\ln (1 - \varepsilon)}{(1 - p + \varepsilon) ^ {2}}.
$$
We therefore need to show that
$$
\frac {- 1}{1 - p + \varepsilon} + \frac {- 1}{p - \varepsilon} \leq \frac {- 1}{(1 - p + \varepsilon) (1 - \varepsilon)} + \frac {- \ln (1 - \varepsilon)}{(1 - p + \varepsilon) ^ {2}}. \tag {C.12}
$$
We note however that
$$
\frac {- 1}{1 - p + \varepsilon} + \frac {- 1}{p - \varepsilon} = \frac {\varepsilon - p - 1 + p - \varepsilon}{(1 - p + \varepsilon) (1 - \varepsilon)} = \frac {- 1}{(1 - p + \varepsilon) (1 - \varepsilon)}.
$$
Therefore Equation (C.12) is equivalent to
$$
\frac {- \ln (1 - \varepsilon)}{(1 - p + \varepsilon) ^ {2}} \geq 0,
$$
which is true since  $\varepsilon \in [0,p]$ . This proves the claim  $x_0^{1 - q} \leq 1 - \varepsilon$ , finishing the proof.
We now prove Lemma C.2 using Lemma C.11 and C.12.
Proof of Lemma C.2. Define the random variable  $Y_{i}$  as  $Y_{i} = X_{i + 1} - q$ . Note that  $Y_{i}$  takes value  $1 - q$  with probability  $p$  and takes  $-q$  with probability  $1 - p$ . Set  $u$  to be the value specified in Lemma C.12. For  $n \geq 0$ , define  $Z_{n} := u^{\sum_{i = 1}^{n}Y_{i}}$ . We first observe that  $Z_{n}$  is a martingale with respect to  $Y_{1},\ldots ,Y_{n}$  as
$$
\begin{array}{l} \mathbb {E} \left[ Z _ {n + 1} \mid Y _ {1}, \dots Y _ {n} \right] = \mathbb {E} \left[ u ^ {\sum_ {i = 1} ^ {n + 1} Y _ {i}} \mid Y _ {1}, \dots Y _ {n} \right] = u ^ {\sum_ {i = 1} ^ {n} Y _ {i}} \cdot (p \cdot u ^ {1 - q} + (1 - p) \cdot u ^ {- q}) \\ = u ^ {\sum_ {i = 1} ^ {n} Y _ {i}} = Z _ {n}. \\ \end{array}
$$
Since  $0 < u < 1$ , this further implies
$$
\begin{array}{l} \Pr \left[ \forall n \geq 0: \sum_ {i = 1} ^ {n} Y _ {i} \geq q - 1 \right] = 1 - \Pr \left[ \exists n \geq 0: \sum_ {i = 1} ^ {n} Y _ {i} <   q - 1 \right] \\ = 1 - \Pr \left[ \max  _ {j \in [ n ]} \left\{u ^ {\sum_ {i = 1} ^ {j} Y _ {i}} \right\} \geq u ^ {q - 1} \right] \\ \geq 1 - \frac {\mathbb {E} [ Z _ {1} ]}{u ^ {q - 1}} \\ = 1 - u ^ {1 - q}, \\ \end{array}
$$
where the first inequality follows from Lemma C.11 and the final equality follows from  $\mathbb{E}\left[Z_1\right] = \mathbb{E}\left[Z_0\right] = \mathbb{E}\left[u^0\right] = 1$
Since  $Y_{i}$  is a function of  $X_{s + 1}$ , we independently have  $X_{1} = 1$  with probability  $p$ . Therefore, with probability  $p(1 - u^{1 - q})$ .
$$
X _ {i} = 1 \text {a n d} \forall n \geq 1: \sum_ {i = 2} ^ {n} (X _ {i} - q) \geq q - 1,
$$
which further implies  $\sum_{i=1}^{n}(X_i - q) \geq 0$ . Therefore,
$$
\operatorname * {P r} \left[ \forall n \geq 1: \frac {\sum_ {i = 1} ^ {n} X _ {i}}{n} \geq q \right] \geq p (1 - u ^ {1 - q}) \geq \Omega (p - q),
$$
where the inequality follows from Equation (C.10).
D Proofs from Section 4: Upper Bounds for Optimistic Agents
D.1 Proof of Theorem 4.1 and Theorem 4.4
We define certain "clean events" to capture desirable realizations of random rewards, and decompose our regret bounds based on whether or not these events hold. The "clean events" ensure that the index of each arm is not too far from its true mean reward; more specifically, that the index is "large enough" for the good arm, and "small enough" for the bad arm. We have two "clean events", one for each arm, defined in terms of the reward-table as follows:
$$
\operatorname {C l e a n} _ {1} ^ {\eta} := \left\{\forall i \in [ T ]: \mathrm {U C B} _ {1, i} ^ {\text {t a p e}, \eta} \geq \mu_ {1} - \Delta / 2 \right\}, \tag {D.1}
$$
$$
\operatorname {C l e a n} _ {2} ^ {\eta} := \left\{\forall i \geq 6 4 \eta / \Delta^ {2}: \mathrm {U C B} _ {2, i} ^ {\text {t a p e}, \eta} \leq \mu_ {2} + \Delta / 4 \right\}. \tag {D.2}
$$
Our analysis is more involved compared to the standard analysis of the UCB1 algorithm [6], essentially because we cannot make  $\eta$  be "as large as needed" to ensure that clean events hold with very high probability. For example, we cannot upper-bound the deviation probability separately for each round and naively take a union bound over all rounds.[20] Instead, we apply a more careful "peeling technique", used e.g., in Audibert and Bubeck [5], so as to avoid any dependence on  $T$  in the lemma below.
20Indeed, this would only guarantee that clean events hold with probability at least  $1 - O(T\cdot e^{-\Omega (\eta)})$  , which in turn would lead to a regret bound like  $O(T^{2}\cdot e^{-\Omega (\eta)})$
Lemma D.1. The clean events hold with probability
$$
\Pr \left[ \operatorname {C l e a n} _ {1} ^ {\eta} \right] \geq 1 - O \left(\left(1 + \log \left(1 / \Delta\right)\right) \cdot e ^ {- \Omega (\eta)}\right), \tag {D.3}
$$
$$
\Pr \left[ \mathrm {C l e a n} _ {2} ^ {\eta} \right] \geq 1 - O \left(e ^ {- \Omega (\eta)}\right). \tag {D.4}
$$
We show that under the appropriate clean events,  $\eta$ -optimistic agents cannot play the bad arm too often. In fact, this claim extends to  $[\eta, \eta_{\max}]$ -optimistic agents.
Claim D.2. Assume that events  $\mathrm{Clean}_1^\eta$  and  $\mathrm{Clean}_2^{\eta_{\mathrm{max}}}$  hold. Then  $[\eta, \eta_{\mathrm{max}}]$ -optimistic agents cannot choose the bad arm more than  $64 \eta_{\mathrm{max}} / \Delta^2$  times.
Proof. For the sake of contradiction, suppose  $[\eta, \eta_{\max}]$ -optimistic agents choose the bad arms at least  $n = 64 \eta_{\max} / \Delta^2$  times, and let  $t$  be the round when this happens. However, by event  $\mathsf{Clean}_1^\eta$ , the index of arm 1 is at least  $\mu_1 - \Delta / 2$ . By event  $\mathsf{Clean}_2^{\eta_{\max}}$ , the index of arm 2 is at least  $\mathsf{UCB}_{i,n}^{\mathrm{tape}, \eta} \leq \mu_2 + \Delta / 4$ , which is less than the index of arm 1, contradiction.
For the "joint" clean event, Clean := Clean $_1^\eta$ $\cap$  Clean $_2^{\eta_{\max}}$ , Lemma D.1 implies
$$
\Pr \left[ \text {C l e a n} \right] \geq 1 - O \left(\log \left(1 / \Delta\right) \cdot e ^ {- \Omega (\eta)}\right). \tag {D.5}
$$
When the clean events fail, we upper-bound regret by  $\Delta \cdot T$ , which is the largest possible. Thus, Lemma D.2 and Eq. (D.5) imply Theorem 4.4, which in turn implies Theorem 4.1 as a special case.
D.2 Proof of Theorem 4.5
We reuse the machinery from Section D.1, but we need some extra work. Recall that all agents are assumed to be  $\eta_{\mathrm{max}}$ -confident, whereas only a fraction are optimistic. Essentially, we rely on the optimistic agents to sample the good arm sufficiently many times (via Claim D.2). Once this happens, all other agents "fall in line" and cannot choose the bad arm too many times.
In what follows, let  $m = 1 + 64\eta_{\mathrm{max}} / \Delta^2$
Claim D.3. Assume Clean. Suppose the good arm is sampled at least  $m$  times by some round  $t_0$ . Then after round  $t_0$ , agents cannot choose the bad arm more than  $m$  times.
Proof. For the sake of contradiction, suppose agent  $t \geq t_0$  has at least  $m$  samples of the bad arm (i.e.,  $n_{2,t} \geq m$ ), and chooses the bad arm once more. Then the index of the good arm satisfies
$$
\begin{array}{l} \operatorname {I n d} _ {1, t} \geq \operatorname {L C B} _ {1, t} ^ {\eta_ {\max }} \quad (\eta_ {\max } - c o n f i d e n t a g e n t s) \\ \geq \operatorname {L C B} _ {1, m} ^ {\text {t a p e}, \eta_ {\max }} \quad (b y \\ \geq \mathrm {U C B} _ {1, m} ^ {\text {t a p e}, \eta_ {\max }} - 2 \sqrt {\eta_ {\max } / m} \quad (\text {b y}) \\ \geq \mathrm {U C B} _ {1, m} ^ {\text {t a p e}, \eta} - 2 \sqrt {\eta_ {\max} / m} \quad (s i n c e \eta_ {\max} \geq \eta) \\ > \mu_ {1} - \Delta / 2 \quad (b y C l e a n _ {1} ^ {\eta} a n d t h e d e f i n i t i o n o f m). \\ \end{array}
$$
The index of the bad arm satisfies
$$
\begin{array}{l} \operatorname {I n d} _ {2, t} \leq \operatorname {U C B} _ {1, t} ^ {\eta} \quad (\eta - \text {c o n f i d e n t a g e n t s}) \\ \leq \mu_ {2} + \Delta / 4 \quad (b y C l e a n _ {1} ^ {\eta} a n d t h e d e f i n i t i o n o f m), \\ \end{array}
$$
which is strictly smaller than  $\operatorname{Ind}_{1,t}$ , contradiction.
For Claim D.3 to "kick in", we need sufficiently many optimistic agents to arrive by time  $t_0$ . Formally, let  $\mathcal{E}_t$  be the event that at least  $2m$  agents are  $[\eta, \eta_{\max}]$ -optimistic in the first  $t$  rounds.
Corollary D.4. Assume Clean. Further, assume event  $\mathcal{E}_{t_0}$  for some round  $t_0$ . Then (by Claim D.2) the good arm is sampled at least  $m$  times before round  $t_0$ . Consequently (by Claim D.3), agents cannot choose the bad arm more than  $m + t_0$  times.
Finally, it is easy to see by Chernoff Bounds that  $\operatorname{Pr}\left[\mathcal{E}_{t_0}\right] \geq 1 - e^{-\Omega(\eta)}$  for some  $t_0 = O(m/q)$  where  $q$  is the probability from the theorem statement. So,  $\operatorname{Pr}\left[\mathrm{Clean} \cap \mathcal{E}_{t_0}\right]$  is lower-bounded as in Eq. (D.5). Again, when  $\mathrm{Clean} \cap \mathcal{E}_{t_0}$  fails, we upper-bound regret by  $\Delta \cdot T$ . So, Corollary D.4 and the lower bound on  $\operatorname{Pr}\left[\mathrm{Clean} \cap \mathcal{E}_{t_0}\right]$  implies the theorem.
D.3 Proof of Lemma D.1
We assume without loss of generality that  $\eta > 2$ . If  $\eta \leq 2$ , the Lemma's statement can be made vacuous using large enough constants in  $O$ . In addition, for mathematical convenience, we will assume that the tape for each arm is infinite, even though the entries after  $T$  will never actually be seen by any of the agents.
For each arm  $a$ , we first separately consider each interval of the form  $[n, 2n]$  and bound the probability that  $\mathrm{UCB}_{a,i}^{\mathrm{tape},\eta}$  deviates too much from  $\mu_{a}$  for  $i \in [n, 2n]$ . While this can be done crudely by applying a union bound over all  $i$ , we use the following maximal inequality.
Lemma D.5 (Eq. (2.17) in [32]). Given a sequence of i.i.d. random variables  $(X_{i})_{i\in [n]}$  in  $[0,1]$  such that  $\mathbb{E}[X_i] = \mu$ , the inequality states that for any  $x > 0$
$$
\operatorname * {P r} \left[ \exists i \in [ n ]: \left| \sum_ {j = 1} ^ {i} (X _ {j} - \mu) \right| > x \right] \leq 2 e ^ {- \frac {2 x ^ {2}}{n}}.
$$
Focusing on some interval of the form  $[n, 2n]$  for  $n \in \mathbb{N}$ , and applying this inequality to the reward tape of arm  $a$ , we conclude that
$$
\Pr \left[ \exists i \in [ n, 2 n ]: \left| \widehat {\mu} _ {a, i} ^ {\text {t a p e}} - \mu_ {a} \right| \geq x \right] \leq O \left(e ^ {- \Omega \left(n x ^ {2}\right)}\right). \tag {D.6}
$$
Define  $f \coloneqq \lceil 64\eta/\Delta^2 \rceil$ . We note that  $f = \Theta(\eta/\Delta^2)$  given the assumption  $\eta > 2$ . In order to bound  $\operatorname{Pr}[\mathrm{Clean}_2^\eta]$ , we will apply this inequality to each interval  $[n, 2n]$  for  $n \geq f$ , and take a union bound. Formally,
$$
\begin{array}{l} 1 - \Pr \left[ \operatorname {C l e a n} _ {2} ^ {\eta} \right] \leq \Pr \left[ \exists i \geq f: \widehat {\mu} _ {2, i} ^ {\text {t a p e}} > \mu_ {2} + \Delta / 8 \right] \quad (S i n c e \sqrt {\eta / i} \leq \Delta / 8 f o r i \geq f) \\ \leq \sum_ {r = 0} ^ {\infty} \Pr \left[ \exists i \in \left[ f 2 ^ {r}, f 2 ^ {r + 1} \right]: \widehat {\mu} _ {2, i} ^ {\text {t a p e}} > \mu_ {2} + \Delta / 8 \right] \quad (U n i o n b o u n d) \\ \leq O \left(\sum_ {r = 0} ^ {\infty} e ^ {- \Omega (\eta 2 ^ {r})}\right) \tag {ByEq.(D.6)} \\ \leq O \left(\sum_ {r = 0} ^ {\infty} e ^ {- \Omega (\eta (r + 1))}\right) \quad (S i n c e 2 ^ {r} \geq r + 1 f o r r \in \mathbb {N}) \\ = O \left(\frac {1}{e ^ {\Omega (\eta)} - 1}\right) \quad (S u m o f g e o m e t r i c s e r i e s) \\ \leq O (e ^ {- \Omega (\eta)}) \quad (B y \eta > 2) \\ \end{array}
$$
In order to bound  $\operatorname{Pr}\left[\mathrm{Clean}_1^\eta\right]$ , we separately handle the intervals  $n < f$  and  $n \geq f$ . For  $n \geq f$ , repeating the same argument as above for arm 1 implies
$$
\Pr \left[ \exists i \geq f: \hat {\mu} _ {1, i} ^ {\text {t a p e}} <   \mu_ {1} - \Delta / 8 \right] \leq O \left(e ^ {- \Omega (\eta)}\right).
$$
For  $n < f$ , we use a modified argument that utilizes the extra  $\sqrt{\eta / i}$  term in  $\mathrm{UCB}_{1,i}^{\mathrm{tape},\eta}$ . Instead of bounding the probability  $\widehat{\mu}_{1,i}^{\mathrm{tape}}$  having deviation  $\Delta /8$ , we bound the probability that it deviates by
$\sqrt{\eta / i}$ . This results in a marked improvement because  $\sqrt{\eta / i}$  increases as we decrease  $i$ . Formally,
$$
\begin{array}{l} \Pr \left[ \exists i \in [ 1, f ]: \widehat {\mu} _ {1, i} ^ {\mathrm {t a p e}} <   \mu_ {1} - \sqrt {\eta / i} \right] \\ \leq \sum_ {r = 0} ^ {\lceil \log (f) \rceil} \Pr \left[ \exists i \in \left[ 2 ^ {r}, 2 ^ {r + 1} \right]: \hat {\mu} _ {1, i} ^ {\text {t a p e}} <   \mu_ {1} - \sqrt {\eta / i} \right] \quad (U n i o n b o u n d) \\ \leq \sum_ {r = 0} ^ {\lceil \log (f) \rceil} \Pr \left[ \exists i \in [ 2 ^ {r}, 2 ^ {r + 1} ]: \hat {\mu} _ {1, i} ^ {\text {t a p e}} <   \mu_ {1} - \sqrt {\eta / 2 ^ {r + 1}} \right] \quad (B y \\ \leq O \left(\sum_ {r = 0} ^ {\lceil \log (f) \rceil} e ^ {- \Omega (\eta)}\right) \tag {ByEq.(D.6)} \\ = O \left(\lceil \log (f) \rceil e ^ {- \Omega (\eta)}\right). \\ \end{array}
$$
Finally, we note that since  $\eta > 2$ ,
$$
\lceil \log (f) \rceil \leq O (1 + \log (f)) = O (1 + \log (\eta) + \log (1 / \Delta)).
$$
This implies Eq. (D.3) because  $O(\log (\eta)e^{-\Omega (\eta)})$  can be rewritten as  $O(e^{-\Omega (\eta)})$  by changing the constant behind  $\Omega$ .
E Proofs from Section 5: Learning Failures for Bayesian Agents
In this section, we prove Theorem 5.1. We first briefly review some properties of the beta distribution. Throughout the section, we consider a beta distribution with parameters  $\alpha, \beta$ .
Lemma E.1 (Fact 1 in [3]). Let  $F_{n,p}^{B}$  denote the CDF of the binomial distribution with parameters  $n, p$  and  $F_{\alpha,\beta}^{\text{beta}}$  denote the CDF of the beta distribution. Then,
$$
F _ {\alpha , \beta} ^ {b e t a} (y) = 1 - F _ {\alpha + \beta - 1, y} ^ {B} (\alpha - 1)
$$
for  $\alpha, \beta$  that are positive integers.
Using Hoeffding's inequality for concentration of the binomial distribution, we immediately obtain the following corollary.
Corollary E.2. Define  $\rho_{\alpha, \beta} \coloneqq \frac{\alpha - 1}{\alpha + \beta - 1}$ . If  $X$  is sampled from the beta distribution with parameters  $(\alpha, \beta)$ ,
$$
\Pr \left[ | X - \rho_ {\alpha , \beta} | \leq y \right] \leq 2 e ^ {- (\alpha + \beta - 1) y ^ {2}}.
$$
In addition, letting  $Q(.)$  denote the quantile function of the distribution,
$$
[ Q (\zeta), Q (1 - \zeta) ] \subseteq \left[ \rho_ {\alpha , \beta} - \sqrt {\frac {\ln (2 / \zeta)}{\alpha + \beta - 1}}, \rho_ {\alpha , \beta} + \sqrt {\frac {\ln (2 / \zeta)}{\alpha + \beta - 1}} \right],
$$
Let  $\alpha_{a,n},\beta_{a,n}$  denote the posterior distribution after observing  $n$  entries of the tape for arm  $a$ . Note that since we are assuming independent priors, the posterior for each arm is independent of the seen rewards of the other arm. Define  $M_{a,n}:= \alpha_{a,n} + \beta_{a,n}$ . We note that by definition,  $\alpha_{a,0},\beta_{a,0}$  coincide with the prior  $\alpha_{a},\beta_{a}$ . We analogously define  $M_{a}:= \alpha_{a} + \beta_{a}$ . Define  $\rho_{a,n}:= \frac{\alpha_{a,n} - 1}{M_{a,n} - 1}$  and  $\xi_{a,n}:= \frac{\alpha_{a,n}}{M_{a,n}}$ . We note that  $\xi_{a,n}$  is the mean of the posterior distribution after observing  $n$  entries of arm  $a$ .
Lemma E.3. For all  $n \geq 0$ ,  $\left|\widehat{\mu}_{a,n}^{\mathrm{tape}} - \xi_{a,n}\right| \leq O\left(\frac{M_{a,0}}{n + M_{a,0}}\right)$ .
Proof. After observing  $n$  entries, the posterior parameters satisfy
$$
\alpha_ {a, n} := \alpha_ {a, 0} + \sum_ {i \leq n} \mathrm {T a p e} _ {a, i}, \quad \beta_ {a, n} := \beta_ {a, 0} + \sum_ {i \leq n} (1 - \mathrm {T a p e} _ {a, i}).
$$
$$
\xi_ {a, n} = \frac {\alpha_ {a , 0} + \sum_ {i \leq n} \mathrm {T a p e} _ {a , i}}{\alpha_ {a , 0} + \beta_ {a , 0} + n}.
$$
Defining  $X := \sum_{i \leq n} \mathsf{Tape}_{a,i}$ , we can bound the difference between  $\xi_{a,n}$  and  $\widehat{\mu}_{a,n}^{\mathrm{tape}}$  as
$$
\begin{array}{l} \left| \frac {\alpha_ {a , 0} + X}{M _ {a , 0} + n} - \frac {X}{n} \right| = \left| \frac {n \alpha_ {a , 0} + n X - n X - X M _ {a , 0}}{n (n + M _ {a , 0})} \right| \\ = \left| \frac {n \alpha_ {a , 0} - X M _ {a , 0}}{n (n + M _ {a , 0})} \right| \\ \leq \frac {\alpha_ {a , 0}}{n + M _ {a , 0}} + \frac {M _ {a , 0}}{n + M _ {a , 0}} \quad (S i n c e X \leq n) \\ \leq O \left(\frac {M _ {a , 0}}{n + M _ {a , 0}}\right) \\ \end{array}
$$
Lemma E.4. For all  $n \geq 0$ ,  $|\xi_{a,n} - \rho_{a,n}| \leq O\left(\frac{1}{n + M_{a,0}}\right)$ .
$$
\begin{array}{l} \left| \frac {\alpha_ {a , n} - 1}{M _ {a , n} - 1} - \frac {\alpha_ {a , n}}{M _ {a , n}} \right| = \left| \frac {- M _ {a , n} + \alpha_ {a , n}}{M _ {a , n} (M _ {a , n} - 1)} \right| \\ \leq \frac {M _ {a , n}}{M _ {a , n} (M _ {a , n} - 1)} \\ = \frac {1}{M _ {a , n} - 1} \\ = O \left(\frac {1}{n + M _ {a , 0}}\right) \quad (S i n c e M _ {a, n} = M _ {a, 0} + n a n d M _ {a, 0} \geq 1) \\ \end{array}
$$
We can now prove Theorem 5.1.
Proof of Theorem 5.1. We start with part (a). Set  $\eta$  to be large enough such that
$$
\left| \hat {\mu} _ {a, n} ^ {\text {t a p e}} - \xi_ {a, n} \right| \leq \sqrt {\frac {\eta}{n}}.
$$