SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kiarash Banihashem
University of Maryland, College Park  
kiarash@umd.edu
MohammadTaghi Hajiaghayi
University of Maryland, College Park hajiagha@umd.edu
Suho Shin
University of Maryland, College Park suhoshin@umd.edu
Aleksandrs Slivkins
Microsoft Research NYC slivkins@microsoft.com
We study social learning dynamics motivated by reviews on online platforms. The agents collectively follow a simple multi-armed bandit protocol, but each agent acts myopically, without regards to exploration. We allow a wide range of myopic behaviors that are consistent with (parameterized) confidence intervals for the arms' expected rewards. We derive stark exploration failures for any such behavior, and provide matching positive results. As a special case, we obtain the first general results on failure of the greedy algorithm in bandits, thus providing a theoretical foundation for why bandit algorithms should explore.<sup>1</sup>
Reviews and ratings are pervasive in many online platforms. A customer consults reviews/ratings, then chooses a product and then (often) leaves feedback, which is aggregated by the platform and served to future customers. Collectively, customers face a tradeoff between exploration and exploitation, i.e., between acquiring new information while making potentially suboptimal decisions and making optimal decisions using information currently available. However, individual customers tend to act myopically and favor exploitation, without regards to exploration for the sake of the others. On a high level, we ask whether/how the myopic behavior interferes with efficient exploration. We are particularly interested in learning failures when only a few agents choose an optimal action.
We distill this issue down to its purest form. We posit that the customers make one decision each and do not observe any personalized payoff-relevant information prior to their decision, whether public or private. In particular, the customers believe they are similar to one another. They have only two alternative products/experiences to choose from, a.k.a., arms, and no way to infer anything about one arm from the other. The platform provides each customer with full history on the previous agents.<sup>2</sup>
Concretely, we posit Bandit Social Learning (BSL): a variant of social learning in which the customers (henceforth, agents) arrive sequentially and follow a simple multi-armed bandit protocol. Each agent observes full history, chooses an arm, and receives a reward: a Bernoulli random draw whose mean is arm-specific and unknown. Initial knowledge (a dataset with some samples of each arm) may be available to all agents. When all agents are governed by a centralized algorithm, this setting is known as stochastic bandits, a standard and well-understood variant of multi-armed bandits.
<sup>1</sup>Early versions of our results on the greedy algorithm (Corollary 3.6 and Theorem 6.1) have been available in a book chapter by A. Slivkins [54, Ch. 11]. The authors acknowledge Mark Sellke for proving Theorem 6.1 and suggesting a proof plan for a version of Corollary 3.6. The authors are grateful to Mark Sellke and Chara Podimata for brief collaborations (with A. Slivkins) in the initial stages of this project.
In practice, online platforms provide summaries such as the average score and the number of samples.
Mean rewards	Beliefs	Behavior	Result
fixed	“frequentist” confidence intervals	η-confident	Thm. 3.1 (main),
Thm. 3.9 (small N0).
unbiased/Greedy	Cor. 3.6
ηt-pessimistic	Thm. 3.10
Bayesian (independent)	Bayesian-unbiased	Thm. 5.1(a)
η-Bayesian-confident	Thm. 5.1(b)
Bayesian (correlated)	Bayesian (and correct)	Bayesian-unbiased	Thm. 6.1
Table 1: Our negative results: learning failures.
We allow a wide range of myopic behaviors that are consistent with observations. Given an arm, consider the confidence interval for its mean reward, parameterized by  $\eta \geq 0$ : the sample average plus/minus the "confidence term",  $\sqrt{\eta / \# \text{samples}}$ . An  $\eta$ -confident agent evaluates each arm to an index: some number within this arm's confidence interval (and otherwise arbitrary), and chooses an arm with a largest index. (Computational implementation of this process is irrelevant to our model.) Crucially, the  $\eta$  and the agents' behavior are given and cannot be influenced by the platform.
This model subsumes the "unbiased" behavior, when the index equals the sample average, as well as various "behavioral biases" (see "related work" for citations). Most notably: "optimism" and "pessimism", when the index is, resp., larger or smaller than the sample average. (These can also be interpreted as, resp., risk seeking and risk aversion.) The model also allows for probabilistic decisions (via randomized indices), correlated behaviors (when samples from one arm affect the behavioral bias on another), and recency bias (when one favors more recent observations). Further, an agent may treat each arm differently, and different agents may exhibit different biases.
We target the regime when parameter  $\eta$  is constant w.r.t. the number of agents  $T$ . I.e., the agents' population is characterized by a constant  $\eta$ . We are interested in the asymptotic behavior when  $T$  increases. An extreme version of our model, with  $\eta \sim \log (T)$ , is only considered for intuition and sanity checks. Interestingly, this version subsumes two well-known bandit algorithms: UCB1 [6] and Thompson Sampling [57, 52], which achieve optimal regret bounds. These algorithms can be seen as behaviors: resp., extreme optimism and probability matching [49, 59], a well-known randomized behavior. More "moderate" versions of these behaviors are consistent with  $\eta$ -confidence as defined above, and are subject to the learning failures described below.
Our results. We are interested in learning failures when all but a few agents choose the bad arm, and how the failure probability scales with the  $\eta$  parameter. Our main result is that with  $\eta$ -confident agents, the failure probability is at least  $e^{-O(\eta)}$  (see Section 3). Consequently, regret is at least  $\Omega(T \cdot e^{-O(\eta)})$  for any given problem instance, in contrast with the  $O(\log T)$  regret rate obtained by optimal bandit algorithms. Further, the  $e^{-O(\eta)}$  scaling is the best possible: indeed, regret for optimistic agents is at most  $O\left(T \cdot e^{-\Omega(\eta)} + \eta\right)$  for a given problem instance (Theorem 4.1). Note that the negative result deteriorates as  $\eta$  increases, and becomes vacuous when  $\eta \sim \log T$ ; the upper bound then essentially matches the optimal  $O(\log T)$  regret of the UCB algorithm [6]).
We refine these results in several directions. First, if all agents are "unbiased", the failure probability scales as the difference in expected reward between the two arms (Corollary 3.6). Second, if all agents are pessimistic, then any level of pessimism, whether small or large or different across agents, leads to the similar failure probability as in the unbiased case (Theorem 3.10). Third, a small fraction of optimists goes a long way! That is, if all agents are  $\eta$ -confident and even a  $q$ -fraction of them are  $\eta$ -optimistic, then we obtain regret  $O\left(T \cdot e^{-\Omega(\eta)} + \eta/q\right)$  regardless of the other agents. $^4$
Our results extend to Bayesian agents who have independent priors across the arms and act according to their posteriors. Such agents are consistent with our main model of  $\eta$ -confident agents, and therefore are subject to the same negative results (Section 5). Further, we focus on Bayesian-unbiased agents and allow arbitrary correlated Bayesian beliefs (when the agents can make inferences about one arm from the observations on the other). We derive a general result on learning failures, assuming that the mean rewards are actually drawn according to the beliefs (Section 6).
${}^{3}$  Then the confidence interval contains the (true) mean reward with probability at least  $1 - 2{e}^{-{2\eta }}$  .
A similar result holds even the agents hold different levels of optimism, see Theorem 4.5.
Mean rewards	Beliefs	Behavior	Result
fixed	“frequentist” confidence intervals	η-optimisticηt-optimistic, ηt ∈ [η, ηmax]small fraction of optimists	Thm. 4.1Thm. 4.4Thm. 4.5
Table 2: Our positive results: upper bounds on regret.
Our results are summarized in Tables 1 and 2.
Implications for multi-armed bandits. The negative results for unbiased agents can be seen as general results on the failure of the greedy algorithm: a bandit algorithm that always exploits. This is a theoretical foundation for why bandit algorithms should explore – and indeed why one should design them. We are not aware of any general results of this nature, whether published or known previously as "folklore", which is quite surprising given the enormous literature on multi-armed bandits. Therefore, we believe our results fill an important gap in the literature.
How surprising are these results? It has been folklore knowledge for several decades that the greedy algorithm is inefficient in some simple special cases, and folklore belief that this should hold much more generally. However, recent results reveal a more complex picture: the greedy algorithm fails under some strong assumptions, but works well under some other strong assumptions (see Related Work). Thus, it has arguably become less clear which assumptions would be needed for negative results and what would be the "shape" and probability of learning failures.
Further, our results on  $\eta$ -confident agents explain why UCB1 algorithm requires extreme optimism, why any algorithm based on narrow (constant- $\eta$ ) confidence intervals is doomed to fail, and also why "pessimism under uncertainty" is not a productive approach for exploration.
Novelty and significance. BSL was not well-understood previously even with unbiased agents, as discussed above, let alone for more permissive behavioral models. It was very unclear a priori how to analyze learning failures and how strong would be the guarantees, in terms of the generality of agents' behaviors, the failure events/probabilities, and the technical assumptions.
On a technical level, our proofs have very little to do with standard lower-bound analyses in bandits stemming from [43, 7]. These analysis apply any algorithm and prove "sublinear" lower bounds on regret, such as  $\Omega (\log T)$  for a given problem instance and  $\Omega (\sqrt{T})$  in the worst case. Their main technical tool is KL-divergence analysis showing that no algorithm can distinguish between a given tuple of "similar" problem instances. In contrast, we prove linear lower bounds on regret, our results apply to a particular family of behaviors/algorithms, and we never consider a tuple of similar problem instances. Instead, we use anti-concentration and martingale tools to argue that the best arm is never played (or played only a few times), with some probability. The result on correlated beliefs in Section 6 has a rather short but "conceptual" proof which we believe is well-suited for a textbook.
While our positive results in Section 4 are restricted to "optimistic" agents, we do not assert that such agents are necessarily typical. The primary point here is that our results on learning failures are essentially tight. That said, "optimism" is a well-documented behavioral bias (e.g., see [50] and references therein). So, a small fraction of optimists (leveraged in Theorem 4.5) is not unrealistic.
Our proofs are more involved compared to the standard analysis of the UCB1 algorithm. This is because we cannot make the  $\eta$  parameter as large as needed to ensure that the complements of certain "clean events" can be ignored. Instead, we need to define and analyze these "clean events" in a more careful way. These difficulties are compounded in Theorem 4.5, our most general result. As far as the statements are concerned, the basic result in Theorem 4.1 is perhaps what one would expect to hold, whereas the extensions in Theorem 4.4 and Theorem 4.5 are more surprising.
Framing. We target the scenario in social learning when both actions and rewards are observable in the future, and the agents do not receive any other payoff-relevant signals. As in much of algorithmic game theory, we discuss the influence of self-interested behavior on the overall welfare of the system. We consider how such behaviour can cause "learning failures", which is a typical framing in the literature on social learning. From the perspective of multi-armed bandits, we investigate the failures of the greedy algorithm, and more generally any algorithm that operates on narrow confidence intervals. We do not attempt to design new algorithms, as a version of UCB1 is proved optimal.
Map of the paper. Section 2 defines our model. Section 3 derives the learning failures. Section 4 provides positive results for optimistic agents. Section 5 and Section 6 handle agents with Bayesian beliefs. Most proofs are moved to the supplement.
Related work. A vast literature on social learning studies agents that learn over time in a shared environment. A prominent topic is learning failures such as ours. Models vary across several dimensions: e.g., the information acquired/transmitted, the communication network, agents' life-span and decision rules, etc. All models from prior work are very different from ours. In the supplement, we separate our model from several most relevant ones: "sequential social learning" [27], "strategic experimentation" [33], networked myopic learners [8, 46], and misspecified beliefs [31, 15, 25, 44].
Positive results for the greedy bandit algorithm [37, 11, 51] focus on contextual bandits, an extension of stochastic bandits where a payoff-relevant signal (context) is available before each round. Equivalently, each agent in BSL receives such signal along with the history (incl. all previous signals). Very strong assumptions are needed: linearity of rewards and diversity of contexts. A similar result holds for BSL with private signals, under different (and also very strong) assumptions on structure and diversity [1]. In all this work, agents' diversity substitutes for exploration, and structural assumptions allow aggregation across agents. Moreover, the greedy algorithm obtains  $o(T)$  regret in various scenarios with a very large number of near-optimal arms [13, 35], e.g., in Bayesian bandits with  $K \gg \sqrt{T}$  arms and independent uniform priors. We focus on a more basic model, with only two arms and no contexts, where all these channels are ruled out.
Learning failures for the greedy algorithm are derived for bandit problems with 1-dimensional action spaces under (strong) structural assumptions: e.g., dynamic pricing with linear demands [30, 22] and dynamic control in a (generalized) linear model [42, 40]. In all these results, the failure probability is only proved positive, but not otherwise characterized.
Incentivized exploration takes a mechanism design perspective on BSL, whereby the platform strives to incentivize individual agents to explore for the sake of the common good. In most of this work, starting from [41, 19], the platform controls the information flow, e.g., can withhold history and instead issue recommendations, and uses this information asymmetry to create incentives; see [55], [54, Ch. 11] for surveys.<sup>5</sup> In particular, [48, 34, 53] target stochastic bandits as the underlying learning problem, same as we do. In [34], the platform constructs a (very) particular communication network for the agents, and then the agents engage in BSL on this network.
Non-Bayesian models of behavior are prominent in social learning, starting from DeGroot [21]: agents use variants of statistical inference and/or naive rules-of-thumb to infer the state of the world. In particular, our model of  $\eta$ -confident agents is essentially a special case of "case-based decision theory" [26]. Well-documented behavioral biases allowed by our model include: optimism and pessimism (e.g., [50] and [18, 12], resp., and references therein), risk aversion/risk seeking [36, 10], recency bias (e.g., [24] and references therein), randomized decisions (with theory tracing back to Luce [47]), and probability matching more specifically [49, 59].
Our perspective of multi-armed bandits is very standard in machine learning theory: we consider asymptotic regret rates without time-doubling. The vast literature on regret-minimizing bandits is summarized in books [17, 54, 45]. Stochastic bandits is a standard, basic version with i.i.d. rewards and no auxiliary structure. Most relevant are the UCB1 algorithm [6], Thompson Sampling [57, 52] (particularly the "frequentist" analyses thereof [2, 4, 38]), and the lower-bound results [43, 7].
2 Our model and preliminaries
Our model, called Bandit Social Learning, is defined as follows. There are  $T$  rounds, where  $T \in \mathbb{N}$  is the time horizon, and two arms (i.e., alternative actions). We use  $[T]$  and [2] to denote the set of rounds and arms, respectively. In each round  $t \in [T]$ , a new agent arrives, observes history  $\mathrm{hist}_t$  (defined below), chooses an arm  $a_t \in [2]$ , receives reward  $r_t \in [0,1]$  for this arm, and leaves forever. When a given arm  $a \in [2]$  is chosen, its reward is drawn independently from Bernoulli distribution
5 Alternatively, the agents observe full history, but the platform uses payments to create incentives [23, 29, 20]. 6 Throughout, we denote  $[n] = \{1,2,\dots ,n\}$  , for any  $n\in \mathbb{N}$
with mean  $\mu_{a} \in [0,1]$ . The mean reward is fixed over time, but not known to the agents. Some initial data is available to all agents, namely  $N_{0} \geq 1$  samples of each arm  $a \in [2]$ . We denote them  $r_{a,i}^{0} \in [0,1], i \in [N_{0}]$ . The history in round  $t$  consists of both the initial data and the data generated by the previous agents. Formally, it is a tuple of arm-reward pairs,
$$
\mathbf {h i s t} _ {t} := \left(\left(a, r _ {a, i} ^ {0}\right): a \in [ 2 ], i \in [ N _ {0} ]; \left(a _ {s}, r _ {s}\right): s \in [ t - 1 ]\right).
$$
We summarize the protocol for Bandit Social Learning as Protocol 1.
Protocol 1: Bandit Social Learning
Problem instance: two arms  $a \in [2]$  with (fixed, but unknown) mean rewards  $\mu_1, \mu_2 \in [0, 1]$ ;
Initialization: hist  $\leftarrow \{N_0$  samples of each arm  $\}$
for each round  $t = 1,2,\dots ,T$  do
agent  $t$  arrives, observes hist and chooses an arm  $a_{t}\in [2]$  reward  $r_t\in [0,1]$  is drawn from Bernoulli distribution with mean  $\mu_{a_t}$  new datapoint  $(a_{t},r_{t})$  is added to hist
Remark 2.1. The initial data-points represent reports created outside our model, e.g., by ghost shoppers or influencers, and available before the products enter the market. One could interpret them as a simple "frequentist" representation for the initial beliefs of the agents, with  $N_0$  as the beliefs' "strength". We posit  $N_0 \geq 1$  to ensure that the arms' average rewards are always well-defined.
If the agents were controlled by an algorithm, this protocol would correspond to stochastic bandits with two arms, the most basic version of multi-armed bandits. A standard performance measure in multi-armed bandits (and online machine learning more generally) is regret, defined as
$$
\operatorname {R e g r e t} (T) := \mu^ {*} \cdot T - \mathbb {E} \left[ \sum_ {t \in [ T ]} \mu_ {a _ {t}} \right], \tag {2.1}
$$
where  $\mu^{*} = \max (\mu_{1},\mu_{2})$  is the maximal expected reward of an arm.
Each agent  $t$  chooses its arm  $a_{t}$  myopically. Each agent is endowed with some (possibly randomized) mapping from histories to arms, and chooses an arm accordingly. This mapping, called behavioral type, encapsulates how the agent resolves uncertainty on the rewards. More concretely, each agent maps the observed history  $\mathrm{hist}_t$  to an index  $\mathrm{Ind}_{a,t} \in \mathbb{R}$  for each arm  $a \in [2]$ , and chooses an arm with a largest index. The ties are broken independently and uniformly at random.
We allow for a range of myopic behaviors, whereby each index can take an arbitrary value in the (parameterized) confidence interval for the corresponding arm. Formally, fix arm  $a \in [2]$  and round  $t \in [T]$ . Let  $n_{a,t}$  denote the number of times this arm has been chosen in the history  $\mathsf{hist}_t$  (including the initial data), and let  $\hat{\mu}_{a,t}$  denote the corresponding average reward. Given these samples, standard (frequentist, truncated) upper and lower confidence bounds for the arm's mean reward  $\mu_a$  (UCB and LCB, for short) are defined as follows:
$$
\mathrm {U C B} _ {a, t} ^ {\eta} := \min  \left\{1, \hat {\mu} _ {a, t} + \sqrt {\eta / n _ {a , t}} \right\} \quad \text {a n d} \quad \mathrm {L C B} _ {a, t} ^ {\eta} := \max  \left\{0, \hat {\mu} _ {a, t} - \sqrt {\eta / n _ {a , t}} \right\}, \tag {2.2}
$$
where  $\eta \geq 0$  is a parameter. The interval  $\left[\mathrm{LCB}_{a,t}^{\eta},\mathrm{UCB}_{a,t}^{\eta}\right]$  will be referred to as  $\eta$ -confidence interval. Standard concentration inequalities imply that  $\mu_{a}$  is contained in this interval with probability at least  $1 - 2e^{-2\eta}$  (where the probability is over the random rewards, for any fixed value of  $\mu_{a}$ ). We allow the index to take an arbitrary value in this interval:
$$
\operatorname {I n d} _ {a, t} \in \left[ \mathrm {L C B} _ {a, t} ^ {\eta}, \mathrm {U C B} _ {a, t} ^ {\eta} \right], \quad \text {f o r e a c h a r m} a \in [ 2 ]. \tag {2.3}
$$
We refer to such agents as  $\eta$ -confident;  $\eta > 0$  will be a crucial parameter throughout.
On special cases. Our model accommodates a number of behavioural biases. Most notably: unbiased agents, who set  $\mathrm{Ind}_{a,t} = \hat{\mu}_{a,t}$ ,  $\eta$ -optimistic agents, who set  $\mathrm{Ind}_{a,t} = \mathrm{UCB}_{a,t}^{\eta}$ , and  $\eta$ -pessimistic agents who set  $\mathrm{Ind}_{a,t} = \mathrm{LCB}_{a,t}^{\eta}$ . Unbiased agents formally correspond to the greedy algorithm, whereas extreme optimism, i.e.,  $\eta$ -optimism with  $\eta \sim \log(T)$ , corresponds to UCB1 algorithm [6].
Our results on upper bounds (Section 4) and Bayesian learning failures (Section 6) allow each arm to have an arbitrary reward distribution on  $[0,1]$ . We omit further mention of this to simplify presentation.
Our model also allows a version of Thompson Sampling in which the posterior samples are truncated to the  $\eta$ -confidence interval. More generally, we allow Bayesian agents that preprocess observations to a Bayesian posterior, and use the latter to define their indices. See the supplement for more details.
Preliminaries. When  $\mu_1, \mu_2$  are fixed (not drawn from a prior), we posit  $\mu_1 > \mu_2$ , i.e., arm 1 is the good arm, and arm 2 is the bad arm. Our guarantees depend on quantity  $\Delta := \mu_1 - \mu_2$ , called the gap (between the two arms). It is a very standard quantity for regret bounds in multi-armed bandits.
We use the big-O notation to hide constant factors. Specifically,  $O(X)$  and  $\Omega(X)$  mean, resp., "at most  $c_0 \cdot X$ " and "at least  $c_0 \cdot X$ " for some absolute constant  $c_0 > 0$  that is not specified in the paper. When and if  $c_0$  depends on some other absolute constant  $c$  that we specify explicitly, we point this out in words and/or by writing, resp.,  $O_c(X)$  and  $\Omega_c(X)$ . As usual,  $\Theta(X)$  is a shorthand for "both  $O(X)$  and  $\Omega(X)$ ", and writing  $\Theta_c(X)$  emphasizes the dependence on  $c$ .
Algorithms UCB1 and Thompson Sampling achieve regret
$$
\operatorname {R e g r e t} (T) \leq O \left(\min  \left(1 / \Delta , \sqrt {T}\right) \cdot \log T\right). \tag {2.4}
$$
This regret rate is essentially optimal among all bandit algorithms: it is optimal up to constant factors for fixed  $\Delta > 0$ , and up to  $O(\log T)$  factors for fixed  $T$  (see "related work" for citations).
A key property of a reasonable bandit algorithm is that  $\mathrm{Regret}(T) / T\to 0$ ; this property is also called no-regret. Conversely, algorithms with  $\mathrm{Regret}(T)\geq \Omega (T)$  are considered very inefficient.
3 Learning failures