SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We are interested in learning failures when all but a few agents choose the bad arm. More precisely, we define the  $n$ -sampling failure as an event that all but at most  $n$  agents choose the bad arm.
We make two technical assumptions:
$$
\text {m e a n} c <   \mu_ {2} <   \mu_ {1} <   1 - c \text {f o r s o m e a b s o l u t e c o n s t a n t} c \in (0, 1 / 2), \tag {3.1}
$$
$$
\text {t h e n u m b e r o f i n i t i a l s a m p l e s s a s i s f i e s} N _ {0} \geq 6 4 \eta / c ^ {2} + 1 / c. \tag {3.2}
$$
The meaning of (3.1) is that it rules out degenerate behaviors when mean rewards are close to the known upper/ lower bounds. The big-O notation hides the dependence on the absolute constant  $c$ , when and if explicitly stated so. Assumption (3.2) ensures that the  $\eta$ -confidence interval is a proper subset of [0, 1] for all agents; we sidestep this assumption later in Theorem 3.9.
Our main result allows arbitrary  $\eta$ -confident agents and asserts that 0-sampling failures happen with probability at least  $p_{\mathrm{fail}} \sim e^{-O(\eta)}$ . This is a stark failure when  $\eta$  is a constant relative to  $T$ .
Theorem 3.1 ( $\eta$ -confident agents). Suppose all agents are  $\eta$ -confident, for some fixed  $\eta \geq 0$ . Make assumptions (3.1) and (3.2). Then the 0-sampling failure occurs with probability at least
$$
p _ {\text {f a i l}} = \Omega_ {c} \left(\Delta + \sqrt {\eta / N _ {0}}\right) \cdot e ^ {- O _ {c} \left(\eta + N _ {0} \Delta^ {2}\right)}, \quad \text {w h e r e} \quad \Delta = \mu_ {1} - \mu_ {2}. \tag {3.3}
$$
Consequently,  $\operatorname{Regret}(T) \geq \Delta \cdot p_{\text{fail}} \cdot T$ .
We emphasize generality: the agents can exhibit any behaviors consistent with  $\eta$ -confidence, possibly different for different agents and different arms. From multi-armed bandit perspective, the theorem implies that bandit algorithms consistent with  $\eta$ -confidence cannot have regret sublinear in  $T$ .
The guarantee in Theorem 3.1 deteriorates as the parameter  $\eta$  increases, and becomes vacuous when  $\eta \sim \log (T)$ . This makes sense, as this regime of  $\eta$  is used in UCB1 algorithm.
Discussion 3.2. Assumption (3.2) is innocuous from the social learning perspective: essentially, the agents hold initial beliefs grounded in data and these beliefs are not completely uninformed. From the bandit perspective, this assumption is more substantive, as an algorithm can always choose to discard data. In any case, we remove this assumption in Theorem 3.9 below.
Remark 3.3. A weaker version of (3.2), namely  $N_0 \geq \eta$ , is necessary to guarantee an  $n$ -sampling failure for any  $\eta$ -confident agents. Indeed, suppose all agents are  $\eta$ -optimistic for arm 1 (the good arm), and  $\eta$ -pessimistic for arm 2 (the bad arm). If  $N_0 < \eta$ , then the index for arm 2 is 0 after the initial samples, whereas the index of arm 1 is always positive. Then all agents choose arm 1.
For  $\eta \sim \log T$ , this coincides with Thompson Sampling with very high probability.
9 Throughout the paper, we use the notation  $O_{c}$  to hide the dependence on the absloute constant  $c$ .
Next, we spell out two corollaries which help elucidate the main result.
Corollary 3.4. If the gap is sufficiently small,  $\Delta < O\left(1 / \sqrt{N_0}\right)$ , then Theorem 3.1 holds with
$$
p _ {\text {f a i l}} = \Omega_ {c} \left(\Delta + \sqrt {\eta / N _ {0}}\right) \cdot e ^ {- O _ {c} (\eta)}. \tag {3.4}
$$
Remark 3.5. The assumption in Corollary 3.4 is quite mild in light of the fact that when  $\Delta > \Omega\left(\sqrt{\log(T) / N_0}\right)$ , the initial samples suffice to determine the best arm with high probability.
Corollary 3.6. If all agents are unbiased, then Theorem 3.1 holds with  $\eta = 0$  and
$$
\begin{array}{l} p _ {\text {f a i l}} = \Omega_ {c} (\Delta) \cdot e ^ {- O _ {c} \left(N _ {0} \Delta^ {2}\right)} \tag {3.5} \\ = \Omega_ {c} (\Delta) \qquad \mathrm {i f} \Delta <   O (1 / \sqrt {N _ {0}}). \\ \end{array}
$$
Remark 3.7. A trivial failure result for unbiased agents relies on the event  $\mathcal{E}$  that all initial samples of arm 1 (i.e., the good arm) are realized as 0. This would indeed imply a 0-sampling failure (as long as at least one initial sample of arm 1 is realized to 1), but the event  $\mathcal{E}$  happens with probability exponential in  $N_0$ , the number of initial samples. In contrast, in our result  $p_{\mathrm{fail}}$  only depends on  $N_0$  through the assumption that  $\Delta < O\left(1 / \sqrt{N_0}\right)$ .
Discussion 3.8. Corollary 3.6 can be seen as a general result on the failure of the greedy algorithm. This is the first such result with a non-trivial dependence on  $N_0$ , to the best of our knowledge.
We can remove assumption (3.2) and allow a small  $N_0$  if the behavioral type for each agent  $t$  also satisfies natural (and very mild) properties of symmetry and monotonicity:
(P1) (symmetry) if all rewards in  $\mathrm{hist}_t$  are 0, the two arms are treated symmetrically, $^{10}$
(P2) (monotonicity) Fix any arm  $a \in [2]$ , any  $t$ -round history  $H$  in which all rewards are 0 for both arms, and any other  $t$ -round history  $H'$  that contains the same number of samples of arm  $a$  such that all these samples have reward 1. Then
$$
\Pr \left[ a _ {t} = a \mid \operatorname {h i s t} _ {t} = H ^ {\prime} \right] \geq \Pr \left[ a _ {t} = a \mid \operatorname {h i s t} _ {t} = H \right]. \tag {3.6}
$$
Note that both properties would still be natural and mild even without the "all rewards are zero" clause. The resulting guarantee on the failure probability is somewhat cleaner.
Theorem 3.9 (small  $N_0$ ). Fix  $\eta \geq 0$ , assume Eq. (3.1), and let  $N_0 \in [1, N^*]$ , where  $N^* := [64\eta / c^2 + 1 / c]$ . Suppose each agent  $t$  is  $\eta$ -confident and satisfies properties (P1) and (P2). Then an  $n$ -sampling failure,  $n = N^* - N_0$ , occurs with probability at least
$$
p _ {\text {f a i l}} = \Omega_ {c} \left(c ^ {2 N ^ {*}}\right) = \Omega_ {c} \left(e ^ {- O _ {c} (\eta)}\right). \tag {3.7}
$$
Consequently,  $\operatorname{Regret}(T) \geq \Delta \cdot p_{\text{fail}} \cdot (T - n)$ .
If all agents are pessimistic, we find that any levels of pessimism, whether small or large or different across agents, lead to a 0-sampling failure with probability  $\Omega_c(\Delta)$ , matching Corollary 3.6 for the unbiased behavior. This happens in the (very reasonable) regime when
$$
\Omega_ {c} (\eta) <   N _ {0} <   O \left(1 / \Delta^ {2}\right). \tag {3.8}
$$
Theorem 3.10 (pessimistic agents). Suppose each agent  $t \in [T]$  is  $\eta_t$ -pessimistic, for some  $\eta_t \geq 0$ . Suppose assumptions (3.1) and (3.2) hold for  $\eta = \max_{t \in [T]} \eta_t$ . Then the 0-sampling failure occurs with probability lower-bounded by Eq. (3.5). Consequently,  $\mathrm{Regret}(T) \geq \Omega_c(\Delta^2) \cdot e^{-O_c(N_0 \Delta^2)}$ .
We allow extremely pessimistic agents  $(\eta_t \sim \log T)$ , and the pessimism levels  $\eta_t$  can vary across agents  $t$ . While the relevant parameter is  $\eta = \max_{t \in [T]} \eta_t$ , the failure probability in (3.5) does not contain the  $e^{-\eta}$  term. In particular,  $p_{\mathrm{fail}} = \Omega(\Delta)$  when  $N_0 < O(1 / \Delta^2)$ . However, the dependence on  $\eta$  "creeps in" through assumption (3.2), i.e., that  $N_0 > \Omega_c(\eta)$ .
Proof Overview. We first show that the average reward of arm 1 (the good arm), is upper bounded by some threshold  $q_{1}$ . This is only guaranteed with some probability and only when this arm is sampled exactly  $N$  times, for a particular  $N \geq N_{0}$ . Next, we lower bound the average reward of arm 2 (the bad arm): we show that with some probability it is always above some threshold  $q_{2} \in (q_{1}, \mu_{2})$ . Focus
That is, the behavioral type stays the same if the arms' labels are switched.
on the round  $t^*$  when the good arm is sampled for the  $N$ -th time (if this ever happens). If both of these events hold, from round  $t^*$  onwards the bad arm will have a larger average reward by a constant margin  $q_2 - q_1$ . Consequently, as we prove, the bad arm has a larger index, and therefore gets chosen.
The details of this argument differ from one theorem to another. For Theorem 3.1, it suffices to set the thresholds  $q_{2}, q_{1}$  such that  $q_{2} - q_{1} = \Theta(\sqrt{\eta / N_{0}})$ . For Theorem 3.10, we use a more involved argument: since the LCB of an arm increases when it is played, playing this arm only strengthens the preference of pessimistic agents for this arm. We are therefore less constrained in the choice of  $q_{1}, q_{2}$  and we can prove that a learning failure occurs whenever  $q_{1} < q_{2}$ . In both proofs, we also require  $q_{1}$  and  $q_{2}$  to be close to  $\mu_{1}$  and  $\mu_{2}$ , resp., so as to lower-bound the probability of the two desirable events. For Theorem 3.9, the case of small  $N_{0}$ , our analysis becomes more subtle. We can (in some sense) simplify the two events defined above, but we need to introduce a third event: if arm 1 is chosen by at least  $n$  agents (for a suitably defined  $n$ ), then arm 2 is chosen by  $n$  agents before arm 1 is. The crux is a "deterministic" argument which derives a failure when all three events hold jointly.
To formalize, we represent realized rewards of each arm  $a$  as written out in advance on a "tape", where each entry is an independent Bernoulli draw with mean  $\mu_{a}$ . The  $i$ -th entry is returned as reward when and if arm  $a$  is chosen for the  $i$ -th time. (We start counting from the initial samples, which comprise entries  $i \in [N_0]$ .) We analyze each arm separately (and then invoke independence).
We use some tools from probability: a sharp anti-concentration inequality for arm 1 and a martingale argument for arm 2. Let  $(X_{i})_{i\in \mathbb{N}}$  be a sequence of i.i.d. Bernoulli random variables with mean  $p\in [c,1 - c]$ , for some absolute constant  $c\in (0,1 / 2)$ . The anti-concentration is as follows:
$$
(\forall n \geq 1 / c, q \in (c / 8, p)) \quad \Pr \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {i} \leq q \right] \geq \Omega \left(e ^ {- O \left(n (p - q) ^ {2}\right)}\right), \tag {3.9}
$$
The martingale argument leads to this:
$$
\forall q \in [ 0, p) \quad \Pr \left[ \forall n \geq 1: \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {i} \geq q \right] \geq \Omega_ {c} (p - q). \tag {3.10}
$$
We each tool to the tape for the respective arm, and lower bound the probability of the desirable event.
While the novelty is mainly in how we use these tools, the tools themselves are not very standard. Eq. (C.1) follows from the anti-concentration inequality in [61] and a reverse Pinsker inequality in [28]. More standard anti-concentration results via Stirling's approximation lead to an additional factor of  $1 / \sqrt{n}$  on the right-hand side of (C.1). For Eq. (C.2), we introduce an exponential martingale and relate the event in Eq. (C.2) to a deviation of this martingale. We then use Ville's inequality (a version of Doob's martingale inequality) to bound the probability that this deviation occurs.
4 Upper bounds for optimistic agents
We upper-bound regret for optimistic agents: we match the exponential-in-  $\eta$  scaling from Corollary 3.4 and then extend this result to different behavioral types. On a technical level, we prove three regret bounds of the same shape (4.1), but with a different  $\Phi$  term. (The unified presentation emphasizes this similarity.) Throughout,  $\Delta = \mu_{1} - \mu_{2}$  denotes the gap between the two arms.
Theorem 4.1. Suppose all agents are  $\eta$ -optimistic, for some fixed  $\eta > 0$ . Then, letting  $\Phi = \eta$
$$
\operatorname {R e g r e t} (T) \leq O \left(T \cdot e ^ {- \Omega (\eta)} \cdot \Delta \left(1 + \log^ {1 / \Delta}\right) + \Phi / \Delta\right). \tag {4.1}
$$
Discussion 4.2. The main take-away is that the exponential-in-  $\eta$  scaling from Corollary 3.4 is tight for  $\eta$ -optimistic agents, and therefore the best possible lower bound for  $\eta$ -confident agents. This result holds for any given  $N_0$ , the number of initial samples. $^{13}$  Our guarantee remains optimal in the "extreme optimism" regime when  $\eta \sim \log (T)$ , matching the optimal regret rate,  $O(\log (T) / \Delta)$ .
What if different agents can hold different behavioral types? First, let us allow agents to have varying amounts of optimism, possibly different across arms and possibly randomized.
Definition 4.3. Fix  $\eta_{\mathrm{max}} \geq \eta > 0$ . An agent  $t \in [T]$  is called  $[\eta, \eta_{\mathrm{max}}]$ -optimistic if its index  $\operatorname{Ind}_{a,t}$  lies in the interval  $\left[\mathsf{UCB}_{a,t}^{\eta}, \mathsf{UCB}_{a,t}^{\eta_{\mathrm{max}}}\right]$ , for each arm  $a \in [2]$ .
<sup>11</sup>We also require  $q_{1} > \sqrt{\eta / N_{0}}$  to ensure that the confidence lower bounds are not truncated to zero.
This is an equivalent (and well-known) representation of rewards in stochastic bandits.
<sup>13</sup>For ease of exposition, we do not track the improvements in regret when  $N_0$  becomes larger.
We show that the guarantee in Theorem 4.1 is robust to varying the optimism level "upwards".
Theorem 4.4 (robustness). Fix  $\eta_{\mathrm{max}} \geq \eta > 0$ . Suppose all agents are  $[\eta, \eta_{\mathrm{max}}]$ -optimistic. Then regret bound (4.1) holds with  $\Phi = \eta_{\mathrm{max}}$ .
Note that the upper bound  $\eta_{\mathrm{max}}$  has only a mild influence on the regret bound in Theorem 4.4.
Our most general result only requires a small fraction of agents to be optimistic, whereas all agents are only required to be  $\eta_{\mathrm{max}}$ -confident (allowing all behaviors consistent with that).
Theorem 4.5 (recurring optimism). Fix  $\eta_{\mathrm{max}} \geq \eta > 0$ . Suppose all agents are  $\eta_{\mathrm{max}}$ -confident. Further, suppose each agent's behavioral type is chosen independently at random so that the agent is  $[\eta, \eta_{\mathrm{max}}]$ -optimistic with probability at least  $q > 0$ . Then regret bound (4.1) holds with  $\Phi = \eta_{\mathrm{max}} / q$ .
Thus, with even a small fraction of optimists,  $q > \frac{1}{\Delta \cdot o(T)}$ , the behavioral type of less optimistic agents does not have much impact on regret. In particular, it does not hurt much if they become very pessimistic. A small fraction of optimists goes a long way! Further, a small-but-constant fraction of extreme optimists, i.e.,  $\eta, \eta_{\max} \sim \log(T)$  in Theorem 4.5, yields optimal regret rate,  $\log(T) / \Delta$ .
5 Learning failures for Bayesian agents
In this section, we posit that agents are endowed with Bayesian beliefs. The basic version is that all agents believe that the mean reward of each arm is initially drawn from a uniform distribution on  $[0,1]$ . (We emphasize that the mean rewards are fixed and not actually drawn according to these beliefs.) Each agent  $t$  computes a posterior  $\mathcal{P}_{a,t}$  for  $\mu_{a}$  given the history  $\mathrm{hist}_t$ , for each arm  $a \in [a]$ , and maps this posterior to the index  $\mathrm{Ind}_{a,t}$  for this arm. $^{14}$
The basic behavior is that  $\mathrm{Ind}_{a,t}$  is the posterior mean reward,  $\mathbb{E}[\mathcal{P}_{a,t}]$ . We call such agents Bayes-unbiased. Further, we consider a Bayesian version of  $\eta$ -confident agents, defined by
$$
\operatorname {I n d} _ {a, t} \in [ Q _ {a, t} (\zeta), Q _ {a, t} (1 - \zeta) ] \quad \text {f o r e a c h a r m} a \in [ 2 ], \tag {5.1}
$$
where  $Q_{a,t}(\cdot)$  denotes the quantile function of the posterior  $\mathcal{P}_{a,t}$  and  $\zeta \in (0,1/2)$  is a fixed parameter (analogous to  $\eta$  elsewhere). The interval in Eq. (5.1) is a Bayesian version of  $\eta$ -confidence intervals. Agents  $t$  that satisfy Eq. (5.1) are called  $\zeta$ -Bayes-confident.
We allow more general beliefs given by independent Beta distributions. For each arm  $a \in [2]$ , all agents believe that the mean reward  $\mu_{a}$  is initially drawn as an independent sample from Beta distribution with parameters  $\alpha_{a}, \beta_{a} \in \mathbb{N}$ . Our results are driven by parameter  $M = \max_{a \in [2]} \alpha_{a} + \beta_{a}$ . We refer to such beliefs as Beta-beliefs with strength  $M$ . The intuition is that the prior on each arm  $a$  can be interpreted as being "based on"  $\alpha_{a} + \beta_{a} - 2$  samples from this arm.[15]
Our technical contribution here is that Bayes-unbiased (resp.,  $\zeta$ -Bayes-confident) agents are  $\eta$ -confident for a suitably large  $\eta$ , and therefore subject to the learning failure in Theorem 3.1.
Theorem 5.1. Consider a Bayesian agent that holds Beta-beliefs with strength  $M \geq 1$ .
(a) If the agent is Bayes-unbiased, then it is  $\eta$ -confident for some  $\eta = O(M / \sqrt{N_0})$ .
(b) If the agent is  $\zeta$ -Bayes-confident, then it is  $\eta$ -confident for  $\eta = O\left(M / \sqrt{N_0} + \ln(1 / \zeta)\right)$ .
Discussion 5.2. Beta-beliefs may be completely unrelated to the actual mean rewards. If  $\zeta$  and  $M$  are constants relative to  $T$ , the resulting  $\eta$  is constant, too. Our guarantee is stronger if the beliefs are weak (i.e.,  $M$  is small) or are "dominated" by the initial samples, in the sense that  $N_0 > \Omega(M^2)$ .
Discussion 5.3.  $\zeta$ -Bayes-confident agents subsume Bayesian version of optimism and pessimism, where the index  $\mathrm{Ind}_{a,t}$  is defined as, resp.,  $Q_{a,t}(1 - \zeta)$  and  $Q_{a,t}(\zeta)$ , as well as all the Bayesian versions of all other behavioral biases discussed previously as special cases of  $\eta$ -confidence.
6 Bayesian model with arbitrary priors
We consider Bayesian-unbiased agents in a "fully Bayesian" model such that the mean rewards are actually drawn from a prior. We are interested in Bayesian probability and Bayesian regret, i.e., resp.,
<sup>14</sup>Note that the Bayesian update for agent  $t$  does not depend on the beliefs of the previous agents.
15More precisely, any Beta distribution with integer parameters  $(\alpha ,\beta)$  can be seen as a Bayesian posterior obtained by updating a uniform prior on [0, 1] with  $\alpha +\beta -2$  data points.
probability and regret in expectation over the prior. We focus on learning failures when the agents never choose an arm with the largest prior mean reward (as opposed to an arm with the largest realized mean reward, which is not necessarily the same arm). We do not explicitly allow initial samples (i.e., we posit  $N_0 = 0$  here), because they are implicitly included in the prior.
Compared to Section 5, the benefit is that we allow arbitrary priors, possibly correlated across the two arms. Further, our guarantee does not depend on the prior, other than through the prior gap  $\mathbb{E}[\mu_1 - \mu_2]$ , and does not contain any hidden constants. On the other hand, the guarantees here are only in expectation over the prior, whereas the ones in Section 5 hold for fixed  $\mu_1, \mu_2$ . Also, our result here is restricted to Bayesian-unbiased agents.
Theorem 6.1. Suppose the pair  $(\mu_1, \mu_2)$  is initially drawn from some Bayesian prior  $\mathcal{P}$  such that  $\mathbb{E}[\mu_1] > \mathbb{E}[\mu_2]$ . Assume that all agents are Bayesian-unbiased, with beliefs given by  $\mathcal{P}$ . Then with Bayesian probability at least  $\mathbb{E}[\mu_1 - \mu_2]$ , the agents never choose arm 2.
Proof. W.l.o.g., assume that agents break ties in favor of arm 2. In each round  $t$ , the key quantity is  $Z_{t} = \mathbb{E}[\mu_{1} - \mu_{2} \mid \mathrm{hist}_{t}]$ . Indeed, arm 2 is chosen if and only if  $Z_{t} \leq 0$ . Let  $\tau$  be the first round when arm 2 is chosen, or  $T + 1$  if this never happens. We use martingale techniques to prove that
$$
\mathbb {E} \left[ Z _ {\tau} \right] = \mathbb {E} \left[ \mu_ {1} - \mu_ {2} \right]. \tag {6.1}
$$
We use the optional stopping theorem (OST). We observe that  $\tau$  is a stopping time relative to  $\mathcal{H} = (\mathsf{hist}_t: t \in [T + 1])$ , and  $(Z_t: t \in [T + 1])$  is a martingale relative to  $\mathcal{H}$ . OST asserts that  $\mathbb{E}[Z_{\tau}] = \mathbb{E}[Z_1]$  for any martingale  $Z_t$  and any bounded stopping time  $\tau$ . Eq. (6.1) follows because  $\mathbb{E}[Z_1] = \mathbb{E}[\mu_1 - \mu_2]$ . On the other hand, by Bayes' theorem it holds that
$$
\mathbb {E} \left[ Z _ {\tau} \right] = \Pr [ \tau \leq T ] \mathbb {E} \left[ Z _ {\tau} \mid \tau \leq T \right] + \Pr [ \tau > T ] \mathbb {E} \left[ Z _ {\tau} \mid \tau > T \right] \tag {6.2}
$$
Recall that  $\tau \leq T$  implies that arm 2 is chosen in round  $\tau$ , which in turn implies that  $Z_{\tau} \leq 0$ . It follows that  $\mathbb{E}[Z_{\tau} \mid \tau \leq T] \leq 0$ . Plugging this into Eq. (6.2), we find that
$$
\mathbb {E} \left[ \mu_ {1} - \mu_ {2} \right] = \mathbb {E} \left[ Z _ {\tau} \right] \leq \Pr \left[ \tau > T \right] = \Pr \left[ \text {a r m 2 n e v e r c h o s e n} \right].
$$
As a corollary, we derive a 0-sampling failure, leading to  $\Omega (T)$  Bayesian regret. Specifically, the agents start out playing arm 1 (because it has a higher prior mean reward), and never try arm 2 when it is in fact the best arm. This happens whenever the prior is independent across arms and has a positive density on the entire [0, 1] interval (see the supplement for the exact statement). Note that it is a (much) more general family of priors compared to independent Beta-priors allowed in Section 5.
7 Conclusions and open questions
We examine the dynamics of social learning in a multi-armed bandit scenario, where agents sequentially choose arms and receive rewards, and observe the full history of previous agents. For a range of agents' myopic behavior, we investigate how they impact exploration, and provide tight upper and lower bounds on the learning failure probabilities and regret rates. In particular, we obtain the first general results on the failure of the greedy algorithm in bandits.
With our results as a "departure point", one could study BSL in more complex bandit models with many arms and/or some known structure of rewards that the agents' myopic behaviour would account for.[17] The greedy algorithm fails for some structures (e.g., our current model) and works well for some others (e.g., for linear contextual bandits with smoothed contexts [37, 11, 51], or when all arms have the same rewards). The whole world is in between these two extremes. It is not at all clear which structures would cause learning failures and which would enable learning, and which structures would be amenable to analysis, one way or another.
This work is partially supported by DARPA QuICC NSF and AF:Small #2218678, #2114269.
The latter follows from a general fact that sequence  $\mathbb{E}[X|\mathrm{hist}_t], t \in [T + 1]$  is a martingale w.r.t.  $\mathcal{H}$  for any random variable  $X$  with  $\mathbb{E}[|X|] < \infty$ . It is known as Doob martingale for  $X$ .
$^{17}E.g.$ , linear, convex, Lipschitz and combinatorial structures are well-studied, see books [17, 54, 45].