Title: Bandit Social Learning under Myopic Behavior
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandits

greedy algorithm

social learning

myopic behavior

learning failures

algorithmic game theory
Abstract: We study social learning dynamics motivated by reviews on online platforms. The
agents collectively follow a simple multi-armed bandit protocol, but each agent
acts myopically, without regards to exploration. We allow a wide range of myopic
behaviors that are consistent with (parameterized) confidence intervals for the arms’
expected rewards. We derive stark exploration failures for any such behavior, and
provide matching positive results. As a special case, we obtain the first general
results on failure of the greedy algorithm in bandits, thus providing a theoretical
foundation for why bandit algorithms should explore.

Bandit Social Learning under Myopic Behavior

Abstract

We study social learning dynamics motivated by reviews on online platforms. The agents collectively follow a simple multi-armed bandit protocol, but each agent acts myopically, without regards to exploration. We allow a wide range of myopic behaviors that are consistent with (parameterized) confidence intervals for the arms' expected rewards. We derive stark exploration failures for any such behavior, and provide matching positive results. As a special case, we obtain the first general results on failure of the greedy algorithm in bandits, thus providing a theoretical foundation for why bandit algorithms should explore. 1

1 Introduction

Reviews and ratings are pervasive in many online platforms. A customer consults reviews/ratings, then chooses a product and then (often) leaves feedback, which is aggregated by the platform and served to future customers. Collectively, customers face a tradeoff between exploration and exploitation , i.e., between acquiring new information while making potentially suboptimal decisions and making optimal decisions using information currently available. However, individual customers tend to act myopically and favor exploitation, without regards to exploration for the sake of the others. On a high level, we ask whether/how the myopic behavior interferes with efficient exploration . We are particularly interested in learning failures when only a few agents choose an optimal action.

We distill this issue down to its purest form. We posit that the customers make one decision each and do not observe any personalized payoff-relevant information prior to their decision, whether public or private. In particular, the customers believe they are similar to one another. They have only two alternative products/experiences to choose from, a.k.a., arms , and no way to infer anything about one arm from the other. The platform provides each customer with full history on the previous agents. 2

Concretely, we posit Bandit Social Learning ( BSL ): a variant of social learning in which the customers (henceforth, agents ) arrive sequentially and follow a simple multi-armed bandit protocol. Each agent observes full history, chooses an arm, and receives a reward: a Bernoulli random draw whose mean is arm-specific and unknown. Initial knowledge (a dataset with some samples of each arm) may be available to all agents. When all agents are governed by a centralized algorithm, this setting is known as stochastic bandits , a standard and well-understood variant of multi-armed bandits.

We allow a wide range of myopic behaviors that are consistent with observations. Given an arm, consider the confidence interval for its mean reward, parameterized by η ≥ 0 : the sample average plus/minus the 'confidence term', √ η/ #samples. 3 An η -confident agent evaluates each arm to an index : some number within this arm's confidence interval (and otherwise arbitrary), and chooses an arm with a largest index. (Computational implementation of this process is irrelevant to our model.) Crucially, the η and the agents' behavior are given and cannot be influenced by the platform.

This model subsumes the 'unbiased' behavior, when the index equals the sample average, as well as various 'behavioral biases' (see 'related work' for citations). Most notably: 'optimism' and 'pessimism', when the index is, resp., larger or smaller than the sample average. (These can also be interpreted as, resp., risk seeking and risk aversion.) The model also allows for probabilistic decisions (via randomized indices), correlated behaviors (when samples from one arm affect the behavioral bias on another), and recency bias (when one favors more recent observations). Further, an agent may treat each arm differently, and different agents may exhibit different biases.

We target the regime when parameter η is constant w.r.t. the number of agents T . I.e., the agents' population is characterized by a constant η . We are interested in the asymptotic behavior when T increases. An extreme version of our model, with η ∼ log( T ) , is only considered for intuition and sanity checks. Interestingly, this version subsumes two well-known bandit algorithms: UCB1 [6] and Thompson Sampling [57, 52], which achieve optimal regret bounds. These algorithms can be seen as behaviors: resp., extreme optimism and probability matching [49, 59], a well-known randomized behavior. More 'moderate' versions of these behaviors are consistent with η -confidence as defined above, and are subject to the learning failures described below.

Our results. We are interested in learning failures when all but a few agents choose the bad arm, and how the failure probability scales with the η parameter. Our main result is that with η -confident agents, the failure probability is at least e -O ( η ) (see Section 3). Consequently, regret is at least Ω( T · e -O ( η ) ) for any given problem instance, in contrast with the O (log T ) regret rate obtained by optimal bandit algorithms. Further, the e -O ( η ) scaling is the best possible: indeed, regret for optimistic agents is at most O ( T · e -Ω( η ) + η ) for a given problem instance (Theorem 4.1). Note that the negative result deteriorates as η increases, and becomes vacuous when η ∼ log T ; the upper bound then essentially matches the optimal O (log T ) regret of the UCB algorithm [6]).

We refine these results in several directions. First, if all agents are 'unbiased', the failure probability scales as the difference in expected reward between the two arms (Corollary 3.6). Second, if all agents are pessimistic, then any level of pessimism, whether small or large or different across agents, leads to the similar failure probability as in the unbiased case (Theorem 3.10). Third, a small fraction of optimists goes a long way! That is, if all agents are η -confident and even a q -fraction of them are η -optimistic, then we obtain regret O ( T · e -Ω( η ) + η/q ) regardless of the other agents. 4

Our results extend to Bayesian agents who have independent priors across the arms and act according to their posteriors. Such agents are consistent with our main model of η -confident agents, and therefore are subject to the same negative results (Section 5). Further, we focus on Bayesian-unbiased agents and allow arbitrary correlated Bayesian beliefs (when the agents can make inferences about one arm from the observations on the other). We derive a general result on learning failures, assuming that the mean rewards are actually drawn according to the beliefs (Section 6).

Our results are summarized in Tables 1 and 2.

Implications for multi-armed bandits. The negative results for unbiased agents can be seen as general results on the failure of the greedy algorithm : a bandit algorithm that always exploits. This is a theoretical foundation for why bandit algorithms should explore - and indeed why one should design them. We are not aware of any general results of this nature, whether published or known previously as "folklore", which is quite surprising given the enormous literature on multi-armed bandits. Therefore, we believe our results fill an important gap in the literature.

How surprising are these results? It has been folklore knowledge for several decades that the greedy algorithm is inefficient in some simple special cases, and folklore belief that this should hold much more generally. However, recent results reveal a more complex picture: the greedy algorithm fails under some strong assumptions, but works well under some other strong assumptions (see Related Work). Thus, it has arguably became less clear which assumptions would be needed for negative results and what would be the 'shape" and probability of learning failures.

Further, our results on η -confident agents explain why UCB1 algorithm requires extreme optimism, why any algorithm based on narrow (constantη ) confidence intervals is doomed to fail, and also why 'pessimism under uncertainty" is not a productive approach for exploration.

Novelty and significance. BSL was not well-understood previously even with unbiased agents, as discussed above, let alone for more permissive behavioral models. It was very unclear a priori how to analyze learning failures and how strong would be the guarantees, in terms of the generality of agents' behaviors, the failure events/probabilities, and the technical assumptions.

On a technical level, our proofs have very little to do with standard lower-bound analyses in bandits stemming from [43, 7]. These analysis apply any algorithm and prove 'sublinear' lower bounds on regret, such as Ω(log T ) for a given problem instance and Ω( √ T ) in the worst case. Their main technical tool is KL-divergence analysis showing that no algorithm can distinguish between a given tuple of 'similar' problem instances. In contrast, we prove linear lower bounds on regret, our results apply to a particular family of behaviors/algorithms, and we never consider a tuple of similar problem instances. Instead, we use anti-concentration and martingale tools to argue that the best arm is never played (or played only a few times), with some probability. The result on correlated beliefs in Section 6 has a rather short but 'conceptual' proof which we believe is well-suited for a textbook.

While our positive results in Section 4 are restricted to 'optimistic' agents, we do not assert that such agents are necessarily typical. The primary point here is that our results on learning failures are essentially tight. That said, 'optimism' is a well-documented behavioral bias ( e.g., see [50] and references therein). So, a small fraction of optimists (leveraged in Theorem 4.5) is not unrealistic.

Our proofs are more involved compared to the standard analysis of the UCB1 algorithm. This is because we cannot make the η parameter as large as needed to ensure that the complements of certain 'clean events' can be ignored. Instead, we need to define and analyze these 'clean events' in a more careful way. These difficulties are compounded in Theorem 4.5, our most general result. As far as the statements are concerned, the basic result in Theorem 4.1 is perhaps what one would expect to hold, whereas the extensions in Theorem 4.4 and Theorem 4.5 are more surprising.

Framing. We target the scenario in social learning when both actions and rewards are observable in the future, and the agents do not receive any other payoff-relevant signals. As in much of algorithmic game theory, we discuss the influence of self-interested behavior on the overall welfare of the system. We consider how such behaviour can cause 'learning failures', which is a typical framing in the literature on social learning. From the perspective of multi-armed bandits, we investigate the failures of the greedy algorithm, and more generally any algorithm that operates on narrow confidence intervals. We do not attempt to design new algorithms, as a version of UCB1 is proved optimal.

Map of the paper. Section 2 defines our model. Section 3 derives the learning failures. Section 4 provides positive results for optimistic agents. Section 5 and Section 6 handle agents with Bayesian beliefs. Most proofs are moved to the supplement.

Related work. A vast literature on social learning studies agents that learn over time in a shared environment. A prominent topic is learning failures such as ours. Models vary across several dimensions: e.g., the information acquired/transmitted, the communication network, agents' life-span and decision rules, etc. All models from prior work are very different from ours. In the supplement, we separate our model from several most relevant ones: 'sequential social learning' [27], 'strategic experimentation' [33], networked myopic learners [8, 46], and misspecified beliefs [31, 15, 25, 44].

Positive results for the greedy bandit algorithm [37, 11, 51] focus on contextual bandits , an extension of stochastic bandits where a payoff-relevant signal ( context ) is available before each round. Equivalently, each agent in BSL receives such signal along with the history (incl. all previous signals). Very strong assumptions are needed: linearity of rewards and diversity of contexts. A similar result holds for BSL with private signals, under different (and also very strong) assumptions on structure and diversity [1]. In all this work, agents' diversity substitutes for exploration, and structural assumptions allow aggregation across agents. Moreover, the greedy algorithm obtains o ( T ) regret in various scenarios with a very large number of near-optimal arms [13, 35], e.g., in Bayesian bandits with K glyph[greatermuch] √ T arms and independent uniform priors. We focus on a more basic model, with only two arms and no contexts, where all these channels are ruled out.

Learning failures for the greedy algorithm are derived for bandit problems with 1-dimensional action spaces under (strong) structural assumptions: e.g., dynamic pricing with linear demands [30, 22] and dynamic control in a (generalized) linear model [42, 40]. In all these results, the failure probability is only proved positive, but not otherwise characterized.

Incentivized exploration takes a mechanism design perspective on BSL , whereby the platform strives to incentivize individual agents to explore for the sake of the common good. In most of this work, starting from [41, 19], the platform controls the information flow, e.g., can withhold history and instead issue recommendations, and uses this information asymmetry to create incentives; see [55], [54, Ch. 11] for surveys. 5 In particular, [48, 34, 53] target stochastic bandits as the underlying learning problem, same as we do. In [34], the platform constructs a (very) particular communication network for the agents, and then the agents engage in BSL on this network.

Non-Bayesian models of behavior are prominent in social learning, starting from DeGroot [21]: agents use variants of statistical inference and/or naive rules-of-thumb to infer the state of the world. In particular, our model of η -confident agents is essentially a special case of 'case-based decision theory' [26]. Well-documented behavioral biases allowed by our model include: optimism and pessimism ( e.g., [50] and [18, 12], resp., and references therein), risk aversion/risk seeking [36, 10], recency bias ( e.g., [24] and references therein), randomized decisions (with theory tracing back to Luce [47]), and probability matching more specifically [49, 59].

Our perspective of multi-armed bandits is very standard in machine learning theory: we consider asymptotic regret rates without time-discounting. The vast literature on regret-minimizing bandits is summarized in books [17, 54, 45]. Stochastic bandits is a standard, basic version with i.i.d. rewards and no auxiliary structure. Most relevant are the UCB1 algorithm [6], Thompson Sampling [57, 52] (particularly the 'frequentist' analyses thereof [2, 4, 38]), and the lower-bound results [43, 7].

2 Our model and preliminaries

Our model, called Bandit Social Learning , is defined as follows. There are T rounds, where T ∈ N is the time horizon, and two arms ( i.e., alternative actions). We use [ T ] and [2] to denote the set of rounds and arms, respectively. 6 In each round t ∈ [ T ] , a new agent arrives, observes history hist t (defined below), chooses an arm a t ∈ [2] , receives reward r t ∈ [0 , 1] for this arm, and leaves forever. When a given arm a ∈ [2] is chosen, its reward is drawn independently from Bernoulli distribution

with mean µ a ∈ [0 , 1] . 7 The mean reward is fixed over time, but not known to the agents. Some initial data is available to all agents, namely N 0 ≥ 1 samples of each arm a ∈ [2] . We denote them r 0 a,i ∈ [0 , 1] , i ∈ [ N 0 ] . The history in round t consists of both the initial data and the data generated by the previous agents. Formally, it is a tuple of arm-reward pairs,

We summarize the protocol for Bandit Social Learning as Protocol 1.

Protocol 1: Bandit Social Learning Problem instance: two arms a ∈ [2] with (fixed, but unknown) mean rewards µ 1 , µ 2 ∈ [0 , 1] ; Initialization: hist ←{ N 0 samples of each arm } ; for each round t = 1 , 2 , . . . , T do agent t arrives, observes hist and chooses an arm a t ∈ [2] ; reward r t ∈ [ 0 , 1 ] is drawn from Bernoulli distribution with mean µ a t ; new datapoint ( a t , r t ) is added to hist

Remark 2.1 . The initial data-points represent reports created outside our model, e.g., by ghost shoppers or influencers, and available before the products enter the market. One could interpret them as a simple 'frequentist' representation for the initial beliefs of the agents, with N 0 as the beliefs' 'strength'. We posit N 0 ≥ 1 to ensure that the arms' average rewards are always well-defined.

If the agents were controlled by an algorithm, this protocol would correspond to stochastic bandits with two arms, the most basic version of multi-armed bandits. A standard performance measure in multi-armed bandits (and online machine learning more generally) is regret , defined as

where µ ∗ = max( µ 1 , µ 2 ) is the maximal expected reward of an arm.

Each agent t chooses its arm a t myopically. Each agent is endowed with some (possibly randomized) mapping from histories to arms, and chooses an arm accordingly. This mapping, called behavioral type , encapsulates how the agent resolves uncertainty on the rewards. More concretely, each agent maps the observed history hist t to an index Ind a,t ∈ R for each arm a ∈ [2] , and chooses an arm with a largest index. The ties are broken independently and uniformly at random.

We allow for a range of myopic behaviors, whereby each index can take an arbitrary value in the (parameterized) confidence interval for the corresponding arm. Formally, fix arm a ∈ [2] and round t ∈ [ T ] . Let n a,t denote the number of times this arm has been chosen in the history hist t (including the initial data), and let ˆ µ a,t denote the corresponding average reward. Given these samples, standard (frequentist, truncated) upper and lower confidence bounds for the arm's mean reward µ a (UCB and LCB, for short) are defined as follows:

where η ≥ 0 is a parameter. The interval [ LCB η a,t , UCB η a,t ] will be referred to as η -confidence interval . Standard concentration inequalities imply that µ a is contained in this interval with probability at least 1 -2 e -2 η (where the probability is over the random rewards, for any fixed value of µ a ). We allow the index to take an arbitrary value in this interval:

We refer to such agents as η -confident ; η > 0 will be a crucial parameter throughout.

On special cases. Our model accommodates a number of behavioural biases. Most notably: unbiased agents , who set Ind a,t = ˆ µ a,t , η -optimistic agents , who set Ind a,t = UCB η a,t , and η - pessimistic agents who set Ind a,t = LCB η a,t . Unbiased agents formally correspond to the greedy algorithm , whereas extreme optimism, i.e., η -optimism with η ∼ log( T ) , corresponds to UCB1 algorithm [6].

Our model also allows a version of Thompson Sampling in which the posterior samples are truncated to the η -confidence interval. 8 More generally, we allow Bayesian agents that preprocess observations to a Bayesian posterior, and use the latter to define their indices. See the supplement for more details.

Preliminaries. When µ 1 , µ 2 are fixed (not drawn from a prior), we posit µ 1 > µ 2 , i.e., arm 1 is the good arm , and arm 2 is the bad arm . Our guarantees depend on quantity ∆ := µ 1 -µ 2 , called the gap (between the two arms). It is a very standard quantity for regret bounds in multi-armed bandits.

We use the big-O notation to hide constant factors. Specifically, O ( X ) and Ω( X ) mean, resp., 'at most c 0 · X ' and 'at least c 0 · X ' for some absolute constant c 0 > 0 that is not specified in the paper. When and if c 0 depends on some other absolute constant c that we specify explicitly, we point this out in words and/or by writing, resp., O c ( X ) and Ω c ( X ) . As usual, Θ( X ) is a shorthand for 'both O ( X ) and Ω( X ) ', and writing Θ c ( X ) emphasizes the dependence on c .

Algorithms UCB1 and Thompson Sampling achieve regret

√

This regret rate is essentially optimal among all bandit algorithms: it is optimal up to constant factors for fixed ∆ > 0 , and up to O (log T ) factors for fixed T (see 'related work' for citations).

A key property of a reasonable bandit algorithm is that Regret ( T ) /T → 0 ; this property is also called no-regret . Conversely, algorithms with Regret ( T ) ≥ Ω( T ) are considered very inefficient.

3 Learning failures

We are interested in learning failures when all but a few agents choose the bad arm. More precisely, we define the n -sampling failure as an event that all but at most n agents choose the bad arm.

We make two technical assumptions:

The meaning of (3.1) is that it rules out degenerate behaviors when mean rewards are close to the known upper/lower bounds. The big-O notation hides the dependence on the absolute constant c , when and if explicitly stated so. Assumption (3.2) ensures that the η -confidence interval is a proper subset of [0 , 1] for all agents; we sidestep this assumption later in Theorem 3.9.

Our main result allows arbitrary η -confident agents and asserts that 0 -sampling failures happen with probability at least p fail ∼ e -O ( η ) . This is a stark failure when η is a constant relative to T .

Theorem 3.1 ( η -confident agents) . Suppose all agents are η -confident, for some fixed η ≥ 0 . Make assumptions (3.1) and (3.2). Then the 0 -sampling failure occurs with probability at least 9

We emphasize generality: the agents can exhibit any behaviors consistent with η -confidence, possibly different for different agents and different arms. From multi-armed bandit perspective, the theorem implies that bandit algorithms consistent with η -confidence cannot have regret sublinear in T .

The guarantee in Theorem 3.1 deteriorates as the parameter η increases, and becomes vacuous when η ∼ log( T ) . This makes sense, as this regime of η is used in UCB1 algorithm.

Discussion 3.2 . Assumption (3.2) is innocuous from the social learning perspective: essentially, the agents hold initial beliefs grounded in data and these beliefs are not completely uninformed. From the bandit perspective, this assumption is more substantive, as an algorithm can always choose to discard data. In any case, we remove this assumption in Theorem 3.9 below.

Remark 3.3 . A weaker version of (3.2), namely N 0 ≥ η , is necessary to guarantee an n -sampling failure for any η -confident agents. Indeed, suppose all agents are η -optimistic for arm 1 (the good arm), and η -pessimistic for arm 2 (the bad arm). If N 0 < η , then the index for arm 2 is 0 after the initial samples, whereas the index of arm 1 is always positive. Then all agents choose arm 1 .

Next, we spell out two corollaries which help elucidate the main result.

Corollary 3.4. If the gap is sufficiently small, ∆ < O ( 1 / √ N 0 ) , then Theorem 3.1 holds with

Remark 3.5 . The assumption in Corollary 3.4 is quite mild in light of the fact that when ∆ > Ω ( √ log( T ) /N 0 ) , the initial samples suffice to determine the best arm with high probability.

Corollary 3.6. If all agents are unbiased, then Theorem 3.1 holds with η = 0 and

Remark 3.7 . A trivial failure result for unbiased agents relies on the event E that all initial samples of arm 1 ( i.e., the good arm) are realized as 0 . This would indeed imply a 0 -sampling failure (as long as at least one initial sample of arm 1 is realized to 1 ), but the event E happens with probability exponential in N 0 , the number of initial samples. In contrast, in our result p fail only depends on N 0 through the assumption that ∆ < O ( 1 / √ N 0 ) .

Discussion 3.8 . Corollary 3.6 can be seen as a general result on the failure of the greedy algorithm. This is the first such result with a non-trivial dependence on N 0 , to the best of our knowledge.

We can remove assumption (3.2) and allow a small N 0 if the behavioral type for each agent t also satisfies natural (and very mild) properties of symmetry and monotonicity:

(P1) (symmetry) if all rewards in hist t are 0 , the two arms are treated symmetrically; 10

(P2) (monotonicity) Fix any arm a ∈ [2] , any t -round history H in which all rewards are 0 for both arms, and any other t -round history H ′ that contains the same number of samples of arm a such that all these samples have reward 1 . Then

Note that both properties would still be natural and mild even without the 'all rewards are zero' clause. The resulting guarantee on the failure probability is somewhat cleaner.

Theorem 3.9 (small N 0 ) . Fix η ≥ 0 , assume Eq. (3.1), and let N 0 ∈ [1 , N ∗ ] , where N ∗ := glyph[ceilingleft] 64 η/c 2 +1 /c glyph[ceilingright] . Suppose each agent t is η -confident and satisfies properties (P1) and (P2). Then an n -sampling failure, n = N ∗ -N 0 , occurs with probability at least

Consequently, Regret ( T ) ≥ ∆ · p fail · ( T -n ) .

If all agents are pessimistic, we find that any levels of pessimism , whether small or large or different across agents, lead to a 0 -sampling failure with probability Ω c (∆) , matching Corollary 3.6 for the unbiased behavior. This happens in the (very reasonable) regime when

Theorem 3.10 (pessimistic agents) . Suppose each agent t ∈ [ T ] is η t -pessimistic, for some η t ≥ 0 . Suppose assumptions (3.1) and (3.2) hold for η = max t ∈ [ T ] η t . Then the 0 -sampling failure occurs with probability lower-bounded by Eq. (3.5). Consequently, Regret ( T ) ≥ Ω c (∆ 2 ) · e -O c ( N 0 ∆ 2 ) .

We allow extremely pessimistic agents ( η t ∼ log T ), and the pessimism levels η t can vary across agents t . While the relevant parameter is η = max t ∈ [ T ] η t , the failure probability in (3.5) does not contain the e -η term. In particular, p fail = Ω(∆) when N 0 < O (1 / ∆ 2 ) . However, the dependence on η 'creeps in' through assumption (3.2), i.e., that N 0 > Ω c ( η ) .

Proof Overview. We first show that the average reward of arm 1 (the good arm), is upper bounded by some threshold q 1 . This is only guaranteed with some probability and only when this arm is sampled exactly N times, for a particular N ≥ N 0 . Next, we lower bound the average reward of arm 2 (the bad arm): we show that with some probability it is always above some threshold q 2 ∈ ( q 1 , µ 2 ) . Focus

on the round t ∗ when the good arm is sampled for the N -th time (if this ever happens). If both of these events hold, from round t ∗ onwards the bad arm will have a larger average reward by a constant margin q 2 -q 1 . Consequently, as we prove, the bad arm has a larger index, and therefore gets chosen.

The details of this argument differ from one theorem to another. For Theorem 3.1, it suffices to set the thresholds q 2 , q 1 such that q 2 -q 1 = Θ( √ η/N 0 ) . For Theorem 3.10, we use a more involved argument: since the LCB of an arm increases when it is played, playing this arm only strengthens the preference of pessimistic agents for this arm. We are therefore less constrained in the choice of q 1 , q 2 and we can prove that a learning failure occurs whenever q 1 < q 2 . 11 In both proofs, we also require q 1 and q 2 to be close to µ 1 and µ 2 , resp., so as to lower-bound the probability of the two desirable events. For Theorem 3.9, the case of small N 0 , our analysis becomes more subtle. We can (in some sense) simplify the two events defined above, but we need to introduce a third event: if arm 1 is chosen by at least n agents (for a suitably defined n ), then arm 2 is chosen by n agents before arm 1 is. The crux is a 'deterministic" argument which derives a failure when all three events hold jointly.

To formalize, we represent realized rewards of each arm a as written out in advance on a 'tape', where each entry is an independent Bernoulli draw with mean µ a . 12 The i -th entry is returned as reward when and if arm a is chosen for the i -th time. (We start counting from the initial samples, which comprise entries i ∈ [ N 0 ] .) We analyze each arm separately (and then invoke independence).

We use some tools from probability: a sharp anti-concentration inequality for arm 1 and a martingale argument for arm 2. Let ( X i ) i ∈ N be a sequence of i.i.d. Bernoulli random variables with mean p ∈ [ c, 1 -c ] , for some absolute constant c ∈ (0 , 1 / 2 ) . The anti-concentration is as follows:

The martingale argument leads to this:

We each tool to the tape for the respective arm, and lower bound the probability of the desirable event.

While the novelty is mainly in how we use these tools, the tools themselves are not very standard. Eq. (C.1) follows from the anti-concentration inequality in [61] and a reverse Pinsker inequality in [28]. More standard anti-concentration results via Stirling's approximation lead to an additional factor of 1 / √ n on the right-hand side of (C.1). For Eq. (C.2), we introduce an exponential martingale and relate the event in Eq. (C.2) to a deviation of this martingale. We then use Ville's inequality (a version of Doob's martingale inequality) to bound the probability that this deviation occurs.

4 Upper bounds for optimistic agents

We upper-bound regret for optimistic agents: we match the exponential-inη scaling from Corollary 3.4 and then extend this result to different behavioral types. On a technical level, we prove three regret bounds of the same shape (4.1), but with a different Φ term. (The unified presentation emphasizes this similarity.) Throughout, ∆ = µ 1 -µ 2 denotes the gap between the two arms.

Theorem 4.1. Suppose all agents are η -optimistic, for some fixed η > 0 . Then, letting Φ = η ,

Discussion 4.2 . The main take-away is that the exponential-inη scaling from Corollary 3.4 is tight for η -optimistic agents, and therefore the best possible lower bound for η -confident agents. This result holds for any given N 0 , the number of initial samples. 13 Our guarantee remains optimal in the 'extreme optimism' regime when η ∼ log( T ) , matching the optimal regret rate, O ( log( T ) / ∆) .

What if different agents can hold different behavioral types? First, let us allow agents to have varying amounts of optimism, possibly different across arms and possibly randomized.

Definition 4.3. Fix η max ≥ η > 0 . An agent t ∈ [ T ] is called [ η, η max ] -optimistic if its index Ind a,t lies in the interval [ UCB η a,t , UCB η max a,t ] , for each arm a ∈ [2] .

We show that the guarantee in Theorem 4.1 is robust to varying the optimism level 'upwards'.

Theorem 4.4 (robustness) . Fix η max ≥ η > 0 . Suppose all agents are [ η, η max ] -optimistic. Then regret bound (4.1) holds with Φ = η max .

Note that the upper bound η max has only a mild influence on the regret bound in Theorem 4.4.

Our most general result only requires a small fraction of agents to be optimistic, whereas all agents are only required to be η max -confident (allowing all behaviors consistent with that).

Theorem 4.5 (recurring optimism) . Fix η max ≥ η > 0 . Suppose all agents are η max -confident. Further, suppose each agent's behavioral type is chosen independently at random so that the agent is [ η, η max ] -optimistic with probability at least q > 0 . Then regret bound (4.1) holds with Φ = η max /q .

Thus, with even a small fraction of optimists, q > 1 ∆ · o ( T ) , the behavioral type of less optimistic agents does not have much impact on regret. In particular, it does not hurt much if they become very pessimistic. A small fraction of optimists goes a long way! Further, a small-but-constant fraction of extreme optimists, i.e., η, η max ∼ log( T ) in Theorem 4.5, yields optimal regret rate, log( T ) / ∆ .

5 Learning failures for Bayesian agents

In this section, we posit that agents are endowed with Bayesian beliefs. The basic version is that all agents believe that the mean reward of each arm is initially drawn from a uniform distribution on [0 , 1] . (We emphasize that the mean rewards are fixed and not actually drawn according to these beliefs.) Each agent t computes a posterior P a,t for µ a given the history hist t , for each arm a ∈ [ a ] , and maps this posterior to the index Ind a,t for this arm. 14

The basic behavior is that Ind a,t is the posterior mean reward, E [ P a,t ] . We call such agents Bayes-unbiased . Further, we consider a Bayesian version of η -confident agents, defined by

where Q a,t ( · ) denotes the quantile function of the posterior P a,t and ζ ∈ (0 , 1 / 2 ) is a fixed parameter (analogous to η elsewhere). The interval in Eq. (5.1) is a Bayesian version of η -confidence intervals. Agents t that satisfy Eq. (5.1) are called ζ -Bayes-confident .

We allow more general beliefs given by independent Beta distributions. For each arm a ∈ [2] , all agents believe that the mean reward µ a is initially drawn as an independent sample from Beta distribution with parameters α a , β a ∈ N . Our results are driven by parameter M = max a ∈ [2] α a + β a . We refer to such beliefs as Beta-beliefs with strength M . The intuition is that the prior on each arm a can be interpreted as being 'based on' α a + β a -2 samples from this arm. 15

Our technical contribution here is that Bayes-unbiased (resp., ζ -Bayes-confident) agents are η -confident for a suitably large η , and therefore subject to the learning failure in Theorem 3.1.

Theorem 5.1. Consider a Bayesian agent that holds Beta-beliefs with strength M ≥ 1 .

(a) If the agent is Bayes-unbiased, then it is η -confident for some η = O ( M/ N 0 ) . √

√

(b) If the agent is ζ -Bayes-confident, then it is η -confident for η = O ( M/ N 0 +ln(1 /ζ ) ) .

Discussion 5.2 . Beta-beliefs may be completely unrelated to the actual mean rewards. If ζ and M are constants relative to T , the resulting η is constant, too. Our guarantee is stronger if the beliefs are weak ( i.e., M is small) or are 'dominated' by the initial samples, in the sense that N 0 > Ω( M 2 ) .

Discussion 5.3 . ζ -Bayes-confident agents subsume Bayesian version of optimism and pessimism, where the index Ind a,t is defined as, resp., Q a,t (1 -ζ ) and Q a,t ( ζ ) , as well as all the Bayesian versions of all other behaviorial biases discussed previously as special cases of η -confidence.

6 Bayesian model with arbitrary priors

We consider Bayesian-unbiased agents in a 'fully Bayesian' model such that the mean rewards are actually drawn from a prior. We are interested in Bayesian probability and Bayesian regret , i.e., resp.,

probability and regret in expectation over the prior. We focus on learning failures when the agents never choose an arm with the largest prior mean reward (as opposed to an arm with the largest realized mean reward, which is not necessarily the same arm). We do not explicitly allow initial samples ( i.e., we posit N 0 = 0 here), because they are implicitly included in the prior.

Compared to Section 5, the benefit is that we allow arbitrary priors, possibly correlated across the two arms. Further, our guarantee does not depend on the prior, other than through the prior gap E [ µ 1 -µ 2 ] , and does not contain any hidden constants. On the other hand, the guarantees here are only in expectation over the prior, whereas the ones in Section 5 hold for fixed µ 1 , µ 2 . Also, our result here is restricted to Bayesian-unbiased agents.

Theorem 6.1. Suppose the pair ( µ 1 , µ 2 ) is initially drawn from some Bayesian prior P such that E [ µ 1 ] > E [ µ 2 ] . Assume that all agents are Bayesian-unbiased, with beliefs given by P . Then with Bayesian probability at least E [ µ 1 -µ 2 ] , the agents never choose arm 2 .

Proof. W.l.o.g., assume that agents break ties in favor of arm 2 . In each round t , the key quantity is Z t = E [ µ 1 -µ 2 | hist t ] . Indeed, arm 2 is chosen if and only if Z t ≤ 0 . Let τ be the first round when arm 2 is chosen, or T +1 if this never happens. We use martingale techniques to prove that

We use the optional stopping theorem (OST). We observe that τ is a stopping time relative to H = ( hist t : t ∈ [ T +1]) , and ( Z t : t ∈ [ T +1]) is a martingale relative to H . 16 OST asserts that E [ Z τ ] = E [ Z 1 ] for any martingale Z t and any bounded stopping time τ . Eq. (6.1) follows because E [ Z 1 ] = E [ µ 1 -µ 2 ] . On the other hand, by Bayes' theorem it holds that

Recall that τ ≤ T implies that arm 2 is chosen in round τ , which in turn implies that Z τ ≤ 0 . It follows that E [ Z τ | τ ≤ T ] ≤ 0 . Plugging this into Eq. (6.2), we find that

As a corollary, we derive a 0-sampling failure, leading to Ω( T ) Bayesian regret. Specifically, the agents start out playing arm 1 (because it has a higher prior mean reward), and never try arm 2 when it is in fact the best arm . This happens whenever the prior is independent across arms and has a positive density on the entire [0 , 1] interval (see the supplement for the exact statement). Note that it is a (much) more general family of priors compared to independent Beta-priors allowed in Section 5.

7 Conclusions and open questions

We examine the dynamics of social learning in a multi-armed bandit scenario, where agents sequentially choose arms and receive rewards, and observe the full history of previous agents. For a range of agents' myopic behavior, we investigate how they impact exploration, and provide tight upper and lower bounds on the learning failure probabilities and regret rates. In particular, we obtain the first general results on the failure of the greedy algorithm in bandits.

With our results as a 'departure point', one could study BSL in more complex bandit models with many arms and/or some known structure of rewards that the agents' myopic behaviour would account for. 17 The greedy algorithm fails for some structures ( e.g., our current model) and works well for some others ( e.g., for linear contextual bandits with smoothed contexts [37, 11, 51], or when all arms have the same rewards). The whole world is in between these two extremes. It is not at all clear which structures would cause learning failures and which would enable learning, and which structures would be amenable to analysis, one way or another.

References

D. Acemoglu, A. Makhdoumi, A. Malekian, and A. Ozdaglar. Learning From Reviews: The Selection Effect and the Speed of Learning. Econometrica , 2022. Working paper available since 2017.

S. Agrawal and N. Goyal. Analysis of Thompson Sampling for the multi-armed bandit problem. In 25nd Conf. on Learning Theory (COLT) , 2012.

S. Agrawal and N. Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory , pages 39-1. JMLR Workshop and Conference Proceedings, 2012.

S. Agrawal and N. Goyal. Near-optimal regret bounds for thompson sampling. J. of the ACM , 64(5):30:1-30:24, 2017. Preliminary version in AISTATS 2013 .

J. Audibert and S. Bubeck. Regret Bounds and Minimax Policies under Partial Monitoring. J. of Machine Learning Research (JMLR) , 11:2785-2836, 2010. Preliminary version in COLT 2009 .

P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47(2-3):235-256, 2002.

P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput. , 32(1):48-77, 2002. Preliminary version in 36th IEEE FOCS , 1995.

V. Bala and S. Goyal. Learning from neighbours. Review of Economic Studies , 65(3):595-621, 1998.

A. V. Banerjee. A simple model of herd behavior. Quarterly Journal of Economics , 107: 797-817, 1992.

N. Barberis and R. Thaler. A survey of behavioral finance. In Handbook of the Economics of Finance , volume 1, pages 1053-1128. Elsevier, 2003.

H. Bastani, M. Bayati, and K. Khosravi. Mostly exploration-free algorithms for contextual bandits. Management Science , 67(3):1329-1349, 2021. Working paper available on arxiv.org since 2017.

M. Bateson. Optimistic and pessimistic biases: a primer for behavioural ecologists. Current Opinion in Behavioral Sciences , 12:115-121, 2016.

M. Bayati, N. Hamidi, R. Johari, and K. Khosravi. Unreasonable effectiveness of greedy algorithms in multi-armed bandit with many arms. In 33rd Advances in Neural Information Processing Systems (NeurIPS) , 2020.

S. Bikhchandani, D. Hirshleifer, and I. Welch. A Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades. Journal of Political Economy , 100(5):992-1026, 1992.

A. Bohren and D. N. Hauser. Learning with Heterogeneous Misspecified Models: Characterization and Robustness. Econometrica , 89(6):3025-3077, Nov 2021.

P. Bolton and C. Harris. Strategic Experimentation. Econometrica , 67(2):349-374, 1999.

S. Bubeck and N. Cesa-Bianchi. Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems. Foundations and Trends in Machine Learning , 5(1):1122, 2012. Published with Now Publishers (Boston, MA, USA). Also available at https://arxiv.org/abs/1204.5721 .

E. C. Chang, editor. Optimism and pessimism: Implications for theory, research, and practice . American Psychological Association, 2000. See https://www.apa.org/pubs/books/431754A for the table of contents.

Y.-K. Che and J. Hörner. Recommender systems as mechanisms for social learning. Quarterly Journal of Economics , 133(2):871-925, 2018. Working paper since 2013, titled 'Optimal design for social learning'.

B. Chen, P. I. Frazier, and D. Kempe. Incentivizing exploration by heterogeneous users. In Conf. on Learning Theory (COLT) , pages 798-818, 2018.

M. H. DeGroot. Reaching a Consensus. Journal of the American Statistical Association , 69 (345):118-121, 1974.

A. V. den Boer and B. Zwart. Simultaneously learning and optimizing using controlled variance pricing. Management Science , 60(3):770-783, 2014.

P. Frazier, D. Kempe, J. M. Kleinberg, and R. Kleinberg. Incentivizing exploration. In ACM Conf. on Economics and Computation (ACM-EC) , pages 5-22, 2014.

D. Fudenberg and D. K. Levine. Learning with recency bias. Proceedings of the National Academy of Sciences , 111:10826-10829, 2014.

D. Fudenberg, G. Lanzani, and P. Strack. Limit Points of Endogenous Misspecified Learning. Econometrica , 89(3):1065-1098, May 2021.

I. Gilboa and D. Schmeidler. Case-Based Decision Theory. The Quarterly Journal of Economics , 110(3):605-639, 08 1995.

B. Golub and E. D. Sadler. Learning in social networks. In Y. Bramoullé, A. Galeotti, and B. Rogers, editors, The Oxford Handbook of the Economics of Networks . Oxford University Press, 2016.

F. Götze, H. Sambale, and A. Sinulis. Higher order concentration for functions of weakly dependent random variables. Electronic Journal of Probability , 24:1-19, 2019.

L. Han, D. Kempe, and R. Qiang. Incentivizing exploration with heterogeneous value of money. In 11th Intl. Conf. on Web and Internet Economics (WINE) , pages 370-383, 2015.

J. M. Harrison, N. B. Keskin, and A. Zeevi. Bayesian dynamic pricing policies: Learning and earning under a binary prior distribution. Management Science , 58(3):570-586, 2012.

P. Heidhues, B. Koszegi, and P. Strack. Unrealistic expectations and misguided learning. Econometrica , 86(4):1159-1214, 2018.

W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American statistical association , 58(301):13-30, 1963.

J. Hörner and A. Skrzypacz. Learning, experimentation, and information design. In B. Honoré, A. Pakes, M. Piazzesi, and L. Samuelson, editors, Advances in Economics and Econometrics: 11th World Congress , volume 1, page 63-98. Cambridge University Press, 2017.

N. Immorlica, J. Mao, A. Slivkins, and S. Wu. Incentivizing exploration with selective data disclosure. In ACM Conf. on Economics and Computation (ACM-EC) , pages 647-648, 2020. Working paper available at https://arxiv.org/abs/1811.06026 .

M. Jedor, J. Louëdec, and V. Perchet. Be greedy in multi-armed bandits, 2021. Working paper, available on https://arxiv.org/abs/2101.01086 .

D. Kahneman and A. Tversky. Judgment under uncertainty: Heuristics and biases . Cambridge University Press, 1982.

S. Kannan, J. Morgenstern, A. Roth, B. Waggoner, and Z. S. Wu. A smoothed analysis of the greedy algorithm for the linear contextual bandit problem. In Advances in Neural Information Processing Systems (NIPS) , pages 2231-2241, 2018.

E. Kaufmann, N. Korda, and R. Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In 23rd Intl. Conf. on Algorithmic Learning Theory (ALT) , pages 199-213, 2012.

G. Keller, S. Rady, and M. Cripps. Strategic Experimentation with Exponential Bandits. Econometrica , 73(1):39-68, 2005.

N. B. Keskin and A. Zeevi. On incomplete learning and certainty-equivalence control. Oper. Res. , 66(4):1136-1167, 2018.

I. Kremer, Y. Mansour, and M. Perry. Implementing the 'wisdom of the crowd". J. of Political Economy , 122(5):988-1012, 2014. Preliminary version in ACM EC 2013 .

T. Lai and H. Robbins. Iterated least squares in multiperiod control. Advances in Applied Mathematics , 3(1):50-73, 1982.

T. L. Lai and H. Robbins. Asymptotically efficient Adaptive Allocation Rules. Advances in Applied Mathematics , 6:4-22, 1985.

G. Lanzani. Dynamic Concern for Misspecification, 2023. Working paper.

T. Lattimore and C. Szepesvári. Bandit Algorithms . Cambridge University Press, Cambridge, UK, 2020.

D. Lazer and A. Friedman. The network structure of exploration and exploitation. Administrative Science Quarterly , 52:667-694, 2007.

R. D. Luce. Individual Choice Behavior: A Theoretical Analysis . Wiley, New York, NY, USA, 1959.

Y. Mansour, A. Slivkins, and V. Syrgkanis. Bayesian incentive-compatible bandit exploration. Operations Research , 68(4):1132-1161, 2020. Preliminary version in ACM EC 2015 .

J. L. Myers. Probability learning and sequence learning. In W. Estes, editor, Handbook of Learning and Cognitive Processes: Approaches to Human Learning and Motivation , pages 171-205. Erlbaum, Hillsdale, NJ, 1976.

M. Puri and D. T. Robinson. Optimism and economic choice. Journal of Financial Economics , 86(1):71-99, 2007.

M. Raghavan, A. Slivkins, J. W. Vaughan, and Z. S. Wu. Greedy algorithm almost dominates in smoothed contextual bandits. SIAM J. on Computing (SICOMP) , 52(2):487-524, 2023. Preliminary version at COLT 2018 . Working paper available at arxiv.org/abs/2005.10624 .

D. Russo, B. Van Roy, A. Kazerouni, I. Osband, and Z. Wen. A tutorial on thompson sampling. Foundations and Trends in Machine Learning , 11(1):1-96, 2018. Published with Now Publishers (Boston, MA, USA). Also available at https://arxiv.org/abs/1707.02038 .

M. Sellke and A. Slivkins. The price of incentivizing exploration: A characterization via thompson sampling and sample complexity. Operations Research , 71(5), 2022. Preliminary version in ACM EC 2021 .

A. Slivkins. Introduction to multi-armed bandits. Foundations and Trends glyph[circleR] in Machine Learning , 12(1-2):1-286, Nov. 2019. Published with Now Publishers (Boston, MA, USA). Also available at https://arxiv.org/abs/1904.07272 . Latest online revision: Jan 2022.

A. Slivkins. Exploration and persuasion. In F. Echenique, N. Immorlica, and V. Vazirani, editors, Online and Matching-Based Market Design . Cambridge University Press, 2023. Also available at http://slivkins.com/work/ExplPers.pdf .

L. Smith and P. Sørensen. Pathological outcomes of observational learning. Econometrica , 68: 371-398, 2000.

W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

J. Ville. Etude critique de la notion de collectif. Bull. Amer. Math. Soc , 45(11):824, 1939.

N. Vulkan. An economist's perspective on probability matching. Journal of Economic Surveys , 14:101-118, 2000.

I. Welch. Sequential sales, learning, and cascades. The Journal of finance , 47:695-732, 1992.

A. R. Zhang and Y. Zhou. On the non-asymptotic and sharp lower tail bounds of random variables. Stat , 9(1):e314, 2020.

SUPPLEMENTARY MATERIALS FOR 'BAYESIAN BANDIT SOCIAL LEARNING UNDER MYOPIC BEHAVIOR'

The supplement contains the proofs, as well as detailed discussion of related work on social learning.

Contents

A Related Work on Social Learning

A vast literature on social learning studies agents that learn over time in a shared environment. A prominent topic is the presence or absence of learning failures such as ours. Models vary across several dimensions, such as: which information is acquired or transmitted, what is the communication network, whether agents are long-lived or only act once, how they choose their actions, etc. Below we discuss several lines of work that are most relevant.

In 'sequential social learning', starting from [9, 60, 14, 56], agents observe private signals, but only the chosen actions are observable in the future; see Golub and Sadler [27] for a survey. The social planner (who chooses agents' actions given access to the knowledge of all previous agents) only needs to exploit , i.e., choose the best action given the previous agents' signals, whereas in our model it also needs to explore . Learning failures are (also) of primary interest, but they occur for an entirely different reason: restricted information flow, i.e., the fact that the private signals are not observable in the future.

'Strategic experimentation', starting from Bolton and Harris [16] and Keller et al. [39], studies long-lived learning agents that observe both actions and rewards of one another; see Hörner and Skrzypacz [33] for a survey. Here, the social planner also solves a version of multi-armed bandits, albeit a very different one (with time-discounting, 'safe' arm that is completely known, and 'risky' arm that follows a stochastic process). The main difference is that the agents engage in a complex repeated game where they explore but prefer to free-ride on exploration by others.

Bala and Goyal [8] and Lazer and Friedman [46] consider a network of myopic learners, all faced with the same bandit problem and observing each other's actions and rewards. The interaction protocol is very different from ours: agents are long-lived, act all at once, and only observe their neighbors on the network. Other specifics are different, too. Bala and Goyal [8] makes strong assumptions on learners' beliefs, which would essentially cause the greedy algorithm to work well in BSL . In Lazer and Friedman [46], each learner only retains the best observed action, rather than the full history. Both papers study social learning under different network topologies.

Prominent recent work, e.g., [31, 15, 25, 44], targets agents with misspecified beliefs , i.e., beliefs whose support does not include the correct model. The framing is similar to BSL with Bayesianunbiased agents: agents arrive one by one and face the same decision problem, whereby each agent makes a rational decision after observing the outcomes of the previous agents. 18 Rational decisions under misspecified beliefs make a big difference compared to BSL , and structural assumptions about rewards/observations and the state space tend to be very different from ours. The technical questions being asked tend to be different, too. E.g., convergence of beliefs is of primary interest, whereas the chosen arms and agents' beliefs/estimates trivially converge in our setting. 19

B Preliminaries: Reward-Tape

It is convenient for our analyses to interpret the realized rewards of each arm as if they are written out in advance on a 'tape'. We posit a matrix ( Tape a,i ∈ [0 , 1] : a ∈ [2] , i ∈ [ T ] ) , called reward-tape , such that each entry Tape a,i is an independent Bernoulli draw with mean µ a . This entry is returned as reward when and if arm a is chosen for the i -th time. (We start counting from the initial samples, which comprise entries i ∈ [ N 0 ] .) This is an equivalent (and well-known) representation of rewards in stochastic bandits.

We will use the notation for the UCBs/LCBs defined by the reward-tape. Fix arm a ∈ [2] and n ∈ [ T ] . Let ̂ µ tape a,n = 1 n ∑ i ∈ [ n ] Tape a,i be the average over the first n entries for arm a . Now, given η ≥ 0 , define the appropriate confidence bounds:

C Proofs from Section 3: Learning Failures

Our proofs rely on two tools from Probability (proved in Section C.4 and C.5): a sharp anticoncentration inequality for Binomial distribution and a lemma that encapsulates a martingale argument.

Lemma C.1 (anti-concentration) . Let ( X i ) i ∈ N be a sequence of independent Bernoulli random variables with mean p ∈ [ c, 1 -c ] , for some c ∈ (0 , 1 / 2 ) interpreted as an absolute constant. Then

where Ω( · ) and O ( · ) hide the dependence on c .

Lemma C.2 (martingale argument) . In the setting of Lemma C.1,

The overall argument will be as follows. We will use Lemma C.1 to upper-bound the average reward of arm 1, i.e., the good arm, by some threshold q 1 . This upper bound will only be guaranteed to hold when this arm is sampled exactly N times, for a particular N ≥ N 0 . Lemma C.2 will allow us to uniformly lower -bound the average reward of arm 2, i.e., the bad arm, by some threshold q 2 ∈ ( q 1 , µ 2 ) . Focus on the round t ∗ when the good arm is sampled for the N -th time (if this ever happens). If the events in both lemmas hold, from round t ∗ onwards the bad arm will have a larger average reward by a constant margin q 2 -q 1 . We will prove that this implies that the bad arm has a larger index, and therefore gets chosen by the agents. The details of this argument differ from one theorem to another.

Lemma C.1 is a somewhat non-standard statement which follows from the anti-concentration inequality in [61] and a reverse Pinsker inequality in [28]. More standard anti-concentration results via Stirling's approximation lead to an additional factor of 1 / √ n on the right-hand side of (C.1). For Lemma C.2, we introduce an exponential martingale and relate the event in (C.2) to a deviation of this martingale. We then use Ville's inequality (a version of Doob's martingale inequality) to bound the probability that this deviation occurs.

C.1 Proof of Theorem 3.1: η -confident agents

Fix thresholds q 1 < q 2 to be specified later. Define two 'failure events':

Fail 1 : the average reward of arm 1 after the N 0 initial samples is below q 1 ;

Fail 2 : the average reward of arm 2 is never below q 2 .

In a formula, using the reward-tape notation from Appendix B, these events are

We show that event Fail := Fail 1 ∩ Fail 2 implies the 0 -sampling failure, as long as the margin q 2 -q 1 is sufficiently large.

Claim C.3. Assume q 2 -q 1 > 2 · √ η/N 0 and event Fail . Then arm 1 is never chosen by the agents.

Proof. Assume, for the sake of contradiction, that some agent chooses arm 1 . Let t be the first round when this happens. Note that Ind 1 ,t ≥ Ind 2 ,t . We will show that this is not possible by upper-bounding Ind 1 ,t and lower-bounding Ind 2 ,t .

By definition of round t , arm 1 has been previously sampled exactly N 0 times. Therefore,

Let n be the number of times arm 2 has been sampled before round t . This includes the initial samples, so n ≥ N 0 . It follows that

Consequently, Ind > Ind

In what follows, let c be the absolute constant from assumption (3.1).

Let us lower bound Pr[ Fail ] by applying Lemmas C.1 and C.2 to the reward-tape.

Claim C.4. Assume c/ 4 < q 1 < q 2 < µ 2 . Then

Proof. To handle Fail 1 , apply Lemma C.1 to the reward-tape for arm 1 , i.e., to the random sequence ( Tape 1 ,i ) i ∈ [ T ] , with n = N 0 and q = q 1 . Recalling that N 0 ≥ 1 /c by assumption (3.2),

To handle Fail 2 , apply Lemma C.2 to the reward-tape for arm 2 , i.e., to the random sequence ( Tape 2 ,i ) i ∈ [ T ] , with threshold q = q 2 . Then

Events Fail 1 and Fail 2 are independent, because they are determined by, resp., realized rewards of arm 1 and realized rewards of arm 2 . The claim follows.

Finally, let us specify suitable thresholds that satisfy the preconditions in Claims C.3 and C.4:

Plugging in µ 2 ≥ c and N 0 ≥ 64 · η/c 2 , it is easy to check that q 1 ≥ c/ 4 , as needed for Claim C.4.

Thus, the preconditions in Claims C.3 and C.4 are satisfied. It follows that the 0 -failure happens with probability at least q fail , as defined in Claim C.4. We obtain the final expression in Eq. (3.3) because µ a -q a = Θ c (∆ + √ η/N 0 ) for both arms a ∈ [2] .

C.2 Proof of Theorem 3.10: pessimistic agents

We reuse the machinery from Section C.1: we define event Fail := Fail 1 ∩ Fail 2 as per Eq. (C.3), for some thresholds q 1 < q 2 to be specified later, and use Claim C.4 to bound Pr[ Fail ] . However, we need a different argument to prove that Fail implies the 0 -sampling failure, and a different way to set the thresholds.

Claim C.5. Assume q 1 > √ η/N 0 and event Fail . Then arm 1 is never chosen by the agents.

Proof. Assume, for the sake of contradiction, that some agent chooses arm 1 . Let t be the first round when this happens. Note that Ind 1 ,t ≥ Ind 2 ,t . We will show that this is not possible by upper-bounding Ind 1 ,t and lower-bounding Ind 2 ,t .

By definition of round t , arm 1 has been previously sampled exactly N 0 times. Therefore,

Let n be the number of times arm 2 has been sampled before round t . This includes the initial samples, so n ≥ N 0 . It follows that

Consequently, Ind 2 ,t > Ind 1 ,t , contradiction.

Now, set the thresholds q 1 , q 2 as follows:

Plugging in µ 2 ≥ c and N 0 ≥ 64 · η/c 2 , it is easy to check that the preconditions in Claims C.4 and C.5 are satisfied. So, the 0 -failure happens with probability at least q fail from Claim C.4. The final expression in Eq. (3.3) follows because µ a -q a = Θ c (∆) for both arms a ∈ [2] .

C.3 Proof of Theorem 3.9: small N 0

We focus on the case when N 0 ≤ N ∗ := glyph[ceilingleft] 64 η/c 2 +1 /c glyph[ceilingright] . We can now afford to handle the initial samples in a very crude way: our failure events posit that all initial samples of the good arm return reward 0 , and all initial samples of the bad arm return reward 1 .

Here, q 2 > 0 is the threshold to be defined later.

On the other hand, our analysis given these events becomes more subtle. In particular, we introduce another 'failure event" Fail 3 , with a more subtle definition: if arm 1 is chosen by at least n := N ∗ -N 0 agents, then arm 2 is chosen by n agents before arm 1 is.

We first show that Fail := Fail 1 ∩ Fail 2 ∩ Fail 3 implies the n -sampling failure.

Claim C.6. Assume that q 2 ≥ c/ 4 and Fail holds. Then at most n = N ∗ -N 0 agents choose arm 1.

Proof. For the sake of contradiction, suppose arm 1 is chosen by more than n agents. Let agent t be the ( n +1) -th agent that chooses arm 1 . In particular, Ind 1 ,t ≥ Ind 2 ,t .

By definition of t , arm 1 has been previously sampled exactly N ∗ times before (counting the N 0 initial samples). Therefore,

Let m be the number of times arm 2 has been sampled before round t . Then

Therefore, Ind 2 ,t > Ind 1 ,t , contradiction.

Next, we lower bound the probability of Fail 1 ∩ Fail 2 using Lemma C.2.

Proof. Instead of analyzing Fail 2 directly, consider events

Note that E ∩E ′ implies Fail 2 . Now, Pr[ Fail 1 ] ≥ µ 1 N ∗ ≥ c N ∗ and Pr[ E ] ≥ (1 -µ 2 ) N ∗ ≥ c N ∗ . Further, Pr[ E ′ ] ≥ Ω c ( µ 2 -q 2 ) by Lemma C.2. The claim follows since these three events are mutually independent.

To bound Pr[ Fail ] , we argue indirectly, assuming Fail 1 ∩ Fail 2 and proving that the conditional probability of Fail 3 is at least 1 / 2 . While this statement feels natural given that Fail 1 ∩ Fail 2 favors arm 2 , the proof requires a somewhat subtle inductive argument. This is where we use the symmetry and monotonicity properties from the theorem statement.

Now, we can lower-bound Pr[ Fail ] by Ω c ( µ 2 -q 2 ) · c 2 N ∗ . Finally, we set the threshold to q 2 = c/ 2 and the theorem follows.

Proof of Claim C.8. Note that event Fail t is determined by the first N ∗ entries of the reward-tape for both arms, in the sense that it does not depend on the rest of the reward-tape.

For each arm a and i ∈ [ T ] , let agent τ a,i be the i -th agent that chooses arm a , if such agent exists, and τ i = T +1 otherwise. Then

Let E be the event that the first N ∗ entries of the reward-tape are 0 for both arms. By symmetry between the two arms (property (P1) in the theorem statement) we have

Next, for two distributions F, G , write F glyph[followsequal] fosd G if F first-order stochastically dominates G . A conditional distribution of random variable X given event E is denoted ( X |E ) . For each i ∈ [ T ] , we consider two conditional distributions for τ 1 ,i : one given Fail 1 ∩ Fail 2 and another given E , and prove that the former dominates:

Applying (C.7) with i = n , it follows that

(The last equality follows from (C.6) and Eq. (C.6).) Thus, it remains to prove (C.7).

Let us consider a fixed realization of each agents' behavioral type, i.e., a fixed, deterministic mapping from histories to arms. W.l.o.g. interpret the behavioral type of each agent t as first deterministically mapping history hist t to a number p t ∈ [0 , 1] , then drawing a threshold θ t ∈ [0 , 1] independently and uniformly at random, and then choosing arm 1 if and only if p t ≥ θ t . Note that p t = Pr[ a t = 1 | hist t ] . So, we pre-select the thresholds θ t for each agent t . Note the agents retain the monotonicity property (P2) from the theorem statement. (For this property, the probabilities on both sides of Eq. (3.6) are now either 0 or 1 .)

Let us prove (C.7) for this fixed realization of the types, using induction on i . Both sides of (C.7) are now deterministic; let A i , B i denote, resp., the left-hand side and the right-hand side. So, we need to prove that A i ≥ B i for all i ∈ [ n ] . For the base case, take i = 0 and define A 0 = B 0 = 0 . For the inductive step, assume A i ≥ B i for some i ≥ 0 . We'd like to prove that A i +1 ≥ B i +1 . Suppose, for the sake of contradiction, that this is not the case, i.e., A i +1 < B i +1 . Since A i < A i +1 by definition of the sequence ( τ a,i : ∈ [ T ]) , we must have

Focus on round t = A i +1 . Note that the history hist t contains exactly i agents that chose arm 1 , both under event Fail 1 ∩ Fail 2 and under event E . Yet, arm 2 is chosen under E , while arm 1 is chosen under Fail 1 ∩ Fail 2 . This violates the monotonicity property (P2) from the theorem statement. Thus, we've proved (C.7) for any fixed realization of the types. Consequently, (C.7) holds in general.

and therefore

C.4 Proof of Lemma C.1

Proof. We use the following sharp lower bound on the tail probability of binomial distribution.

Theorem C.9 (Theorem 9 in [61]) . Let n ∈ N be a positive integer and let ( X i ) i ∈ [ n ] be a sequance of i.i.d Bernoulli random variables with prameter p . For any β > 1 there exists constants c β and C β that only rely on β , such that for all x satisfying x ∈ [0 , np β ] and x + n (1 -p ) ≥ 1 , we have

where D ( x || y ) denotes the KL divergence between two Bernoulli random variables with parameters x and y .

We use the above result with x = n ( p -q ) and β = 1 -c 1 -9 8 c . Note that β > 1 since c < 1 2 . We first verify that x, β satisfy the conditions of the lemma. The x + n (1 -p ) ≥ 1 condition holds by the assumption n ≥ 1 /c :

As for the x ≤ np β condition, by definition of x ,

Since p ≤ 1 -c and p p -q is decreasing in p for p ≥ q , we can further bound this with

where the second inequality follows from q ≥ c/ 8 and q < p ≤ 1 -c , together with the fact that 1 -c 1 -c -q is decreasing in q for q < 1 -c . We obtain x ≤ np β by rearranging.

Next, we use the following type of reverse Pinsker's inqeuality to upper bound D ( q || p ) .

Theorem C.10 ([28]) . For any two probability measures P and Q on a finite support X , if Q is absolutely continuous with respect to P , then the their KL divergence D ( Q || P ) is upper bounded by 2 α P δ ( Q,P ) 2 where α P = min x ∈ X P ( x ) and δ ( Q,P ) denotes the total variation distance between P and Q .

Setting P = Bernoulli ( p ) and Q = Bernoulli ( q ) , we have α P = min( p, 1 -p ) , and δ ( Q,P ) = p -q Therefore, since min( p, 1 -p ) ≥ c by assumption, we conclude D ( q || p ) ≤ O (( p -q ) 2 ) . Plugging this back in Equation (C.8) finshes the proof.

C.5 Proof of Lemma C.2

Our proof will rely on the following doob-style inequality for (super)martingales.

Lemma C.11 (Ville's Inequality [58]) . Let ( Z n ) n ≥ 0 be a positive supermartingale with respect to filtration ( F n ) n ≥ 0 , i.e. Z n ≥ E [ Z n +1 |F n ] for any n ≥ 0 . Then the following holds for any x > 0 ,

In order to use this result, we will define the martingale Z n := u ∑ n i =1 ( X i +1 -q ) for a suitable choice of u as specified by the following lemma.

Lemma C.12. Let c be an absolute constant. For any p ∈ ( c, 1 -c ) and q ∈ (0 , p ) , there exists a value of u ∈ (0 , 1) such that

Proof. To see why such a u exists, define f ( x ) = ( p · x 1 -q +(1 -p ) · x -q ) . It is clear that f (1) = 1 and lim x → 0 f ( x ) = ∞ as lim x → 0 (1 -p ) x -q = ∞ . Furthermore,

which implies

Therefore, f ( x ) is decreasing at x = 1 . Since lim x → 0 f ( x ) > f (1) , this implies that f ( u ) = f (1) for some u ∈ (0 , 1) , proving Equation (C.9).

We now prove Equation (C.10), define x 0 as x 0 = (1 -p ) q p (1 -q ) . Note that x 0 < 1 since p > q . We claim that u ≤ x 0 . To see why, we first note that f ′ ( x ) can be rewritten as

It is clear that f ′ ( x 0 ) = 0 . Since xp (1 -q ) -(1 -p ) q is increasing in x , this further implies that f ′ ( x ) > 0 for x > x 0 . Now, if u > x 0 , then since f ′ ( x ) > 0 for x > x 0 , we would conclude that f ( u ) < f (1) , which is not possible since f ( u ) = f (1) = 1 . Therefore, u ≤ x 0 as claimed.

We now claim that x 1 -q 0 ≤ 1 -p + q . This would finish the proof since, together with u ≤ x 0 , this would imply

where for the last equation we have used the assumption p ∈ ( c, 1 -c ) .

To prove the claim, define ε := p -q . We need to show that x 1 -q 0 ≤ 1 -ε , or equivalently ln( x 0 ) ≤ ln(1 -ε ) 1 -q . By defintion of x 0 , this is equivalent to

Fix p and consider both hand sides as a function of ε . Putting ε = 0 , both hands side coincide as they both equal 0. To prove Euqation (C.11), it suffices to show that as we increase ε , the left hand side decreases faster than the right hand side. Equivalently, we need to show that the derivative of the LHS with respect to ε is larger than the derivative of the RHS with respect to ε for ε ≤ [0 , p ] . Taking the derivative with respect to ε on LHS, we obtain

Similarly taking the derivative on RHS we obtain

We therefore need to show that

We note however that

Therefore Equation (C.12) is equivalent to

which is true since ε ∈ [0 , p ] . This proves the claim x 1 -q 0 ≤ 1 -ε , finishing the proof.

In addition, u satisfies

We now prove Lemma C.2 using Lemma C.11 and C.12.

proof of Lemma C.2. Define the random variable Y i as Y i = X i +1 -q . Note that Y i takes value 1 -q with probability p and takes -q with probability 1 -p . Set u to be the value specified in Lemma C.12. For n ≥ 0 , define Z n := u ∑ n i =1 Y i . We first observe that Z n is a martingale with respect to Y 1 , . . . , Y n as

Since 0 < u < 1 , this further implies

where the first inequality follows from Lemma C.11 and the final equality follows from E [ Z 1 ] = E [ Z 0 ] = E [ u 0 ] = 1 .

Since Y i is a function of X s +1 , we independently have X 1 = 1 with probability p . Therefore, with probability p (1 -u 1 -q ) .

which further implies ∑ n i =1 ( X i -q ) ≥ 0 . Therefore,

where the inequality follows from Equation (C.10).

D Proofs from Section 4: Upper Bounds for Optimistic Agents

D.1 Proof of Theorem 4.1 and Theorem 4.4

We define certain 'clean events' to capture desirable realizations of random rewards, and decompose our regret bounds based on whether or not these events hold. The 'clean events' ensure that the index of each arm is not too far from its true mean reward; more specifically, that the index is 'large enough' for the good arm, and 'small enough' for the bad arm. We have two 'clean events', one for each arm, defined in terms of the reward-table as follows:

Our analysis is more involved compared to the standard analysis of the UCB1 algorithm [6], essentially because we cannot make η be 'as large as needed' to ensure that clean events hold with very high probability. For example, we cannot upper-bound the deviation probability separately for each round and naively take a union bound over all rounds. 20 Instead, we apply a more careful 'peeling technique', used e.g., in Audibert and Bubeck [5], so as to avoid any dependence on T in the lemma below.

Lemma D.1. The clean events hold with probability

We show that under the appropriate clean events, η -optimistic agents cannot play the bad arm too often. In fact, this claim extends to [ η, η max ] -optimistic agents.

Claim D.2. Assume that events Clean η 1 and Clean η max 2 hold. Then [ η, η max ] -optimistic agents cannot choose the bad arm more than 64 η max / ∆ 2 times.

Proof. For the sake of contradiction, suppose [ η, η max ] -optimistic agents choose the bad arms at least n = 64 η max / ∆ 2 times, and let t be the round when this happens. However, by event Clean η 1 , the index of arm 1 is at least µ 1 -∆ / 2 . By event Clean η max 2 , the index of arm 2 is at least UCB tape , η i,n ≤ µ 2 +∆ / 4 , which is less than the index of arm 1, contradiction.

For the 'joint' clean event, Clean := Clean η 1 ∩ Clean η max 2 , Lemma D.1 implies

When the clean events fail, we upper-bound regret by ∆ · T , which is the largest possible. Thus, Lemma D.2 and Eq. (D.5) imply Theorem 4.4, which in turn implies Theorem 4.1 as a special case.

D.2 Proof of Theorem 4.5

We reuse the machinery from Section D.1, but we need some extra work. Recall that all agents are assumed to be η max -confident, whereas only a fraction are optimistic. Essentially, we rely on the optimistic agents to sample the good arm sufficiently many times (via Claim D.2). Once this happens, all other agents 'fall in line' and cannot choose the bad arm too many times.

In what follows, let m = 1 + 64 η max / ∆ 2 .

Claim D.3. Assume Clean . Suppose the good arm is sampled at least m times by some round t 0 . Then after round t 0 , agents cannot choose the bad arm more than m times.

Proof. For the sake of contradiction, suppose agent t ≥ t 0 has at least m samples of the bad arm ( i.e., n 2 ,t ≥ m ), and chooses the bad arm once more. Then the index of the good arm satisfies

The index of the bad arm satisfies

which is strictly smaller than Ind 1 ,t , contradiction.

For Claim D.3 to 'kick in', we need sufficiently many optimistic agents to arrive by time t 0 . Formally, let E t be the event that at least 2 m agents are [ η, η max ] -optimistic in the first t rounds.

Corollary D.4. Assume Clean . Further, assume event E t 0 for some round t 0 . Then (by Claim D.2) the good arm is sampled at least m times before round t 0 . Consequently (by Claim D.3), agents cannot choose the bad arm more than m + t 0 times.

Finally, it is easy to see by Chernoff Bounds that Pr[ E t 0 ] ≥ 1 -e -Ω( η ) for some t 0 = O ( m/q ) , where q is the probability from the theorem statement. So, Pr[ Clean ∩ E t 0 ] is lower-bounded as in Eq. (D.5). Again, when Clean ∩ E t 0 fails, we upper-bound regret by ∆ · T . So, Corollary D.4 and the lower bound on Pr[ Clean ∩ E t 0 ] implies the theorem.

D.3 Proof of Lemma D.1

We assume without loss of generality that η > 2 . If η ≤ 2 , the Lemma's statement can be made vacuous using large enough constants in O . In addition, for mathematical convenience, we will assume that the tape for each arm is infinite, even though the entries after T will never actually be seen by any of the agents.

For each arm a , we first separately consider each interval of the form [ n, 2 n ] and bound the probability that UCB tape , η a,i deviates too much from µ a for i ∈ [ n, 2 n ] . While this can be done crudely by applying a union bound over all i , we use the following maximal inequality.

Lemma D.5 (Eq. (2.17) in [32]) . Given a sequence of i.i.d. random variables ( X i ) i ∈ [ n ] in [0 , 1] such that E [ X i ] = µ , the inequality states that for any x > 0 ,

Focusing on some interval of the form [ n, 2 n ] for n ∈ N , and applying this inequality to the reward tape of arm a , we conclude that

Define f := glyph[ceilingleft] 64 η/ ∆ 2 glyph[ceilingright] . We note that f = Θ( η/ ∆ 2 ) given the assumption η > 2 . In order to bound Pr[ Clean η 2 ] , we will apply this inequality to each interval [ n, 2 n ] for n ≥ f , and take a union bound. Formally,

In order to bound Pr[ Clean η 1 ] , we separately handle the intervals n < f and n ≥ f . For n ≥ f , repeating the same argument as above for arm 1 implies

For n < f , we use a modified argument that utilizes the extra √ η/i term in UCB tape , η 1 ,i . Instead of bounding the probability ̂ µ tape 1 ,i having deviation ∆ / 8 , we bound the probability that it deviates by

√ η/i . This results in a marked improvement because √ η/i increases as we decrease i . Formally,

Finally, we note that since η > 2 ,

This implies Eq. (D.3) because O (log( η ) e -Ω( η ) ) can be rewritten as O ( e -Ω( η ) ) by changing the constant behind Ω .

E Proofs from Section 5: Learning Failures for Bayesian Agents

In this section, we prove Theorem 5.1. We first briefly review some properties of the beta distribution. Throughout the section, we consider a beta distribution with parameters α, β .

Lemma E.1 (Fact 1 in [3]) . Let F B n,p denote the CDF of the binomial distribution with paramters n, p and F beta α,β denote the CDF of the beta distribution. Then,

for α, β that are positive integers.

Using Hoeffding's inequality for concentration of the binomial distribution, we immediately obtain the following corollary.

Corollary E.2. Define ρ α,β := α -1 α + β -1 . If X is sampled from the beta distribution with parameters ( α, β ) ,

In addition, letting Q ( . ) denote the quantile function of the distribution,

Let α a,n , β a,n denote the posterior distribution after observing n entries of the tape for arm a . Note that since we are assuming independent priors, the posterior for each arm is independent of the seen rewards of the other arm. Define M a,n := α a,n + β a,n . We note that by definition, α a, 0 , β a, 0 coincide with the prior α a , β a . We analogously define M a := α a + β a . Define ρ a,n := α a,n -1 M a,n -1 and ξ a,n := α a,n M a,n . We note that ξ a,n is the mean of the posterior distribution after observing n entries of arm a .

Proof. After observing n entries, the posterior parameters satisfy

It follows that

Defining X := ∑ i ≤ n Tape a,i , we can bound the difference between ξ a,n and ̂ µ tape a,n as

Lemma E.4. For all n ≥ 0 , | ξ a,n -ρ a,n | ≤ O ( 1 n + M a, 0 ) .

Proof.

We can now prove Theorem 5.1.

Proof of Theorem 5.1. We start with part (a). Set η to be large enough such that

Since M a n + M a ≤ M a n , by Lemma E.3, this can be achieved with η ≥ O ( M a / √ N 0 ) , which proves part (a).

For part (b), set η to be large enough such that ∣ ∣ ̂ µ tape a,n -ρ a,n ∣ ∣ ≤ 1 2 · √ η n . Given, Lemmas E.3 and E.4, this can be achieved with η ≥ O ( M a / √ N 0 ) . Since M -1 ≥ n , we can further gaurantee ln(2 /ζ ) M -1 ≤ η 4 n by setting η ≥ O (ln(1 /ζ )) , which finishes the proof together with Corollary E.2.

F The corollary from Section 6

As a corollary of Theorem 6.1, we derive a 0-sampling failure, leading to Ω( T ) Bayesian regret. Specifically, the agents never try arm 2 when it is in fact the best arm . This happens whenever the prior is independent across arms and has a positive density on the entire [0 , 1] interval.

Corollary F.1. In the setting of Theorem 6.1, suppose the prior P is independent across arms and has a positive density for each arm ( i.e., has probability density function that is strictly positive on [0 , 1] ). Then E [ Regret ( T )] ≥ c P · T , where the constant c P > 0 depends only on the prior P .

This follows from a more explicit, but more cumbersome corollary.

Corollary F.2. Denote µ 0 1 = E [ µ 1 ] and µ 0 2 = E [ µ 2 ] . Consider independent priors such that Pr[ µ 1 = 1] < ( µ 0 1 -µ 0 2 ) / 2 . Pick any α > 0 such that Pr[ µ 1 ≥ 1 -2 α ] ≤ ( µ 0 1 -µ 0 2 ) / 2 . Then Bayesian regret is at least T · ( α / 2 ( µ 0 1 -µ 0 2 ) Pr [ µ 2 > 1 -α ] ) .

Proof. Let E 1 be the event that µ 1 < 1 -2 α and arm 2 is never chosen. By Theorem 6.1 and the definition of α , we have Pr[ E 1 ] ≥ ( µ 0 1 -µ 0 2 ) / 2 .

Let E 2 be the event that µ 2 > 1 -α . Under event E 1 ∩ E 2 , each round contributes µ 2 -µ 1 ≥ α to regret, so E [ R ( T ) | E 1 ∩ E 2 ] ≥ αT .

Since event E 1 is determined by the prior on arm 1 and the rewards of arm 2 , it is independent from E 2 . It follows that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers social learning in a two-armed Bernoulli bandit scenario, where agents sequentially arrive and pull an arm with the highest index, where the index is arbitrarily chosen to be within some confidence bound of the empirical mean of the arm, parametrized by $\\eta$. This behavior subsumes greedy behavior ($\\eta = 0$) and regret-optimal policies such as UCB1. The main contribution is a tight characterization of the probability of learning failure, i.e., most agents will pull the suboptimal arm as a function of the reward gap and $\\eta$. They extend this result to the case where agents are Bayesian and use truncated priors to inform indices.\n\nSTRENGTHS:\n- As the paper suggests, while the fact that greedy behavior causes learning failure with constant probability in standard classes of K armed bandit problems is folklore, there is no formal study of the boundaries of regimes where greedy behavior starts failing. Thus the motivation of the paper is solid.\n- The techniques for proving the lower bounds on the probabilities of failure are novel relative to standard bandit literature.\n\nWEAKNESSES:\n- The paper's conceptual takeaways are not surprising, and the technical contribution adds little beyond tightly characterizing the probabilities of failure as a function of $\\eta$, the value of which is unclear for such a specific model of social learning. \n- The examples where the greedy behavior suffices in prior literature are contextual bandit environments where context diversity is the driver of exploration. So it seems that understanding the boundaries of greedy behavior must work with some interpolation between contextual and independent armed environments. Instead, the present paper focuses on a two-armed bandit environment where the confidence bounds are parametrized, the value of which is unclear since one anticipates learning failure for any fixed parameter value. \n- The interpretation of the results for the non-Bayesian setting is obfuscated by the dependency on $N_0$, the initial number of samples. It seems that a lot of technical maneuvering (e.g., assumption 3.2) arises because, when $N_0$ is small, one cannot eliminate the possibility of avoiding learning failure due to the confidence bounds being truncated at the boundary of $[0,1]$. This seems orthogonal to the central issue of focus (in this regard, Theorem 3.9 certainly appears to be cleaner). This makes the results appear too technical without adding anything substantial to the dialogue on the sufficiency of greedy algorithms for bandit learning.\n\nQUESTIONS:\nCould you comment on the qualitative difference between Theorem 3.9 and Theorem 3.1, and whether focusing on Theorem 3.9 with the very reasonable assumptions in P1-P2 would achieve the goals of the paper?\n\n---------------------------\nPost rebuttal: Thanks for the clarifications. While my original concerns about the practicality of regimes and the surprise-factor remain, I see the conceptual value of the results and believe they deserve to be published. I have raised my score accordingly.\n\nLIMITATIONS:\nThe results pertain to a very specific two-armed bandit model and so the paper is explicit about its limitations and applicability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper posits a bandit social learning (BSL) problem, which consists of a multi-armed bandit (MAB) problem where at each round an arm is pulled by a newly arrived agent, as a function of the history. This is motivated by reviews on online platforms, where agents pick decisions sequentially based on past reviews. Compared to standard MAB, where a centralized algorithm is run to minimize regret, in BSL each agent acts myopically and can be, e.g., greedy, optimistic, or pessimistic w.r.t. confidence intervals constructed around the reward estimate for each arm. The authors analyze the 2-arms setup and provide several learning failure results in identifying the optimal arm. Notably, the learning \"fails\" when agents are greedy or pessimistic, while it achieves optimal regret when a small fraction of the agents are optimistic. This was a general belief in standard MAB, but to the best of authors' knowledge their results are the first ones to assess it theoretically. Similar learning failures are also established for Bayesian agents who act according to their posterior.\n\nSTRENGTHS:\n- The paper reads really well and the results are sound. Moreover, the authors did a good job of positioning it into the existing related literature.\n- The introduced BSL problem is simple, albeit very interesting, and of practical relevance, e.g., in review systems. \n- Although the flavor of the results is quite specific to the BSL problem (where agents have different myopic behavior), the negative results apply to the more general MAB, constituting a relevant contribution to the broader bandits community.\n\nWEAKNESSES:\n- The authors consider 2-armed bandits for the sake of their analysis and negative results. However, it is not clear how the picture would change in the presence of more arms.\n- No experiments are performed. Although the paper is of theoretical nature, would be nice to demonstrate the proven failure probabilities and how the injected optimism facilitates learning.\n\nQUESTIONS:\nI would like to know the author's view on extending such results to more than 2 arms. In particular, is a bigger set of arms always more detrimental in terms of learning/exploration? Should the initial number of samples $N_0$ intuitively scale with the number of arms?\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper studies bandit social learning problem with two arms. Instead of aiming to design an efficient algorithm with theoretial guarantee, the authors demonstrate negative results regarding myoptic behaviors of agents. The main contribution of this paper is proving the regret lower bounds of $\eta$-confidence agents, together with nearly matching upper bounds, which explains why greedy algorithms i.e., always exploit, are not efficient and why UCB1 algorithm requires extreme optimism.

STRENGTHS:
* This paper is well written and easy to follow. The author does a great job of explaining complex concepts in a clear and concise way.
* The proofs appear solid and complete to me that both of UCB and Bayesian agents are taken into consideration in this paper.

WEAKNESSES:
* I believe this is a good paper because it provides solid theoretical insights into why agents perform less optimistically as UCB-type algorithms perform worse, and how this degradation varies with the degree of optimism. However, I feel that the proofs do not introduce any new techniques, which lowers my overall score.

QUESTIONS:
It might be better to add experiment simulations to validate the theory.

*Typo*

Line 39: the number of agents $T$...: should here be number of time steps?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-02

comments: SUMMARY:
The paper proposes the model of social learning under myopic behavior, where a 2-armed bandit problem is considered with agents that behave myopically. Upper and lower bounds on the probability that all but \leq n agents choose the bad arm are derived.

STRENGTHS:
- The paper is well written and easy to follow.
- The considered model is novel.
- The bounds on failure probability are tight.

WEAKNESSES:
- The results are only limited to the case of 2 arms.
- Assumption 3.2 is strong and seems to be unnecessary. For example if mu_1 is very small, then a lot of initial samples N0 are required even if mu_2 is close to 1 which is an easy to solve case that does not require a lot of samples.
- The considered myopic strategies are limited to a few number of strategies (confident, unbiased, optimistic, pessimistic, bayesian).
- The results are not surprising and can be obtained using standard concentration inequalities in the bandit literature, e.g., see [1].
- As the paper is only concerned about failure probabilities without the need to decide on a strategy, I am not convinced about the importance of the results.

[1] Lattimore, Tor, and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a social learning problem motivated by reviews on online platforms, where (myopic) users make (purchase) decisions based on historical reviews and generate new reviews in an online fashion. It considers several different user behavioral types, such as the confidence based optimistic, pessimistic and neutral users. In addition, the users could have Bayesian belief on the mean reward and act according to the posterior update based on the history. The paper characterizes several cases where the learning failure occurs i.e., when all but a few agents choose the bad arm.\n\nSTRENGTHS:\n1. This paper introduces and analyzes an interesting setup of social learning. I can see how this setup may be applied to many real world applications, such as online recommendation systems (especially in presence of purchase decisions). I also expect many research projects to follow up on this fundamental social learning model.\n2. The technical tools used in this paper are somewhat different from the standard bandit literature.\n3. The possible results as the implications regarding the optimistic agents are very interesting.\n\nWEAKNESSES:\n1. The presentation and structure of this paper requires some improvements. The current version seems to have many results scattered around the paper, making it hard for us readers to switch the context from one section to another.  Section 3 is named as \"learning failure\", but it actually considers several different setups (theorem 3.1, 3.9, 3.10). Why not merge theorem 3.1 and 3.9? It might also be a good idea to use a table to summarize the results under different setups, e.g., when it fails, when it doesn't.  \n2. Overall, I find the problem setting interesting, but the technical results are not that surprising. For example, Theorem 3.1 requires a fixed \\eta for every agent, which more or less violates its motivation of social learning --- the agents tend to have heterogeneous behavior types. I also wish the paper could connect its results and setups to real world applications (instead of imagining purely idealized scenarios).\n3. The entire paper chooses to only focus on the two arms case. I expect the author to point out the exact technical challenges (or practical motivations) that prevent the analysis to be extended to the general cases.\n\nQUESTIONS:\nPlease see the points listed up the weakness part. I really liked the problem setup, and I am willing to raise my score if the authors could convince me of the technical/conceptual significance of their existing results.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-16"}