Diversity-Preserving $K$--Armed Bandits, Revisited
Authors: Hedi Hadiji
Sébastien Gerchinovitz
Jean-Michel Loubes
Gilles Stoltz
Venue: Accepted by TMLR ()

Abstract: We consider the bandit-based framework for diversity-preserving recommendations introduced by Celis et al. (2019), who approached it in the case of a polytope mainly by a reduction to the setting of linear bandits. We design a UCB algorithm using the specific structure of the setting and show that it enjoys a bounded distribution-dependent regret in the natural cases when the optimal mixed actions put some probability mass on all actions (i.e., when diversity is desirable). The regret lower bounds provided show that otherwise, at least when the model is mean-unbounded, a $\ln T$ regret is suffered. We also discuss an example beyond the special case of polytopes.

Published in Transactions on Machine Learning Research (07/2024)
Diversity-Preserving K-Armed Bandits, Revisited
Hédi Hadiji
heddhhadijj@cccnoolssseee...
L2S CNRS CentroleSupéêec - Université Paris-Saclay, Gif-sur- Yvette
Sébastien Gerchinovitz
Institut de recherche technologique Saint Exupéry, Toulouse Instifut de mathématiques de Toulouse, Université Paul Sabatier, Toulouse
Jean-Michel Loubes
Institut de mathématiques de Toulouse, Université Paud Sabatier, Toulouse
Gilles Stoltz
gilles. surtl::@uuiiversiieppriiss---
Université Paris-Saclay, CNRS, Laboratoire de mathématiques d' d'Orsay, Orsay, France
HEC Paris, Jouy-en-Josas, France
Reviewed on OpenReview: https: // opw.eevee.........oomo. net/ forun? id- Vuz7KBqO4A
Abstract
We consider the bandit-based framework for diversity-preserinng recommendations intro- duced by Celis et al. (2019), who approached it the case of a polytope mainly by a reduction the setting of linear bandits. We design UCB algorithm using the specific structure of the sctting and show that it enjoys bounded distribution-deeender regret in the natural cases when the optimal mixed actions put some probability mass all actions (i.e., when diversity is desirable). The regret lower bounds provided show that otherwise, at least when the model is mean-unbounded In T regret is suffered. We also discuss an example beyond the special case of polytopes.
1 Introduction
Ensuring fairness recommendation systems has been a major concern in the machine learning literature among the last years, due to the growing influence of recommender systems in all parts our socicties. If many definitions of fairness have been provided, here we focus on models that ensure minimum allocation guarantees. Hence fairness has be understood in terms of eeversity-preserviig constraints, as stated in the recent EU Artificial Intelligence Acti] which imposes that AI recommender system should preserve the fundamental right. of equal access to opportunities. We refer to Silva et al. (2022), Li et al. (2021) Huang et al. (2021). and references therein, for some recent reviews on this topic. We will consider a bandit- based framework to model such recommendation algorithms,, as introduced by the seminal work of Celis ct al. 2019). The fairness framework presented in this study addresses a variety of scenarios involving the distribution of resources. In the context of search engine advertisements, it mandates that every advertiser is allocated a specified share of advertisement impressions, thereby preventing any single entity from dominating the available ad space. Similarly, in task planning platforms, ensures that every participant receives proportionate number of tasks, which is erucial to avoid disparate treatments. Furthermore, within wireless communication systems, this model obligates the receiver to maintain baseline level of service quality for every transmitter, guaranteeing that all senders receive equitable treatment. This approach not only promotes fairness but also enhances the overall system's 's effectiveness by fostering diversity and participation across different settings. We refer to Patil et al. 2021 or Molina et al. 2023) for different methods promoting fairness in this context.
httper//aattficcaiiieeellsgeeee

---

Published in Transactions on Machine Learning Research (07/2024)
The setting by Celis et al. (2019). We consider stochastic K-armed bandits. All arms correspond to desirable actions or options, though some lead to higher payoffs. Effective (regret-minimizing are bound to play the optimal arm(s) an overwhelming fraction of time. Celis et al. (2019) refer to this effect as polarization (see details in Example D and introduce the learning protocol of Section 1.1 to avoid it. In a nutshell, this protocol consists of picking arms A, in two-slage randomization, by choosing first a probability distribution P, over the arms within a sct P of admissible distributions, and then by drawing the arm A to be actually played according to this distribution P. The simplest example of admissible distributions is composed (see Example I of distributions all putting at least some minimal probability mass l > 0 on each arm. All actions or options corresponding to the arms then get some fair chance to be selected, even if their expected payoffs are far from the expected payoff of the optimal arms. We therefore suggest the alternative terminology of preserving diversity. The two-stage randomization considered makes sense in scenarios where there is a strong internal commitment to respect diversity but randomization is needed to respect privacy, or where some central authority oventually picks the arms based on the distributions output. Additional justifications of the setting are provided by Celis et al. (2019)
algorithms
Overview of our results. Our aim in this article is to deepen the thcoretical results obtained by Colis
et al. 2019 see more details in Section 1.2 In a nutshell, Celis et al. (019)) approached the problem described above for polytopes P and by reducing it to the case of linear bandits: by ignoring that actions A, are eventually played and by only considering the distributions P, chosen. Doing so, they obtain suboptimal rates for regret bounds. In particular, we show that bounded regret may be achieved in the favorable cases where diversity is actually desired: for sub-Gaussian models and when the optimal admissible distributions put some positive probability mass on all arms (which happens, in particular, when the set P is bounded away from the boundary of the simplex of probability distributions). Actually, at least in mean-unbounded sub-Gaussian models, we even characterize bounded regret as a feature of optimal distributions putting some probability mass on all arms. We also consider a case where P not a polytope but a set with a continuous and curved set of extremal points. In that case, the reduction to linear bandits would provide vT regret bounds while we show that using the information given by A., the regret may grow only at a squared-logarithmmt rate, again showing a stark improvement.
A case of special interest (see Example 2) is a version of bandits with knapsacks where constraints are to be satisfied in expectation at all rounds, and thus, by martingale convergence, also almost surely in the limit. The associated theory is, of course, of a fundamentally simpler nature than the hard constraints typically considered in the litcrature of bandits with knapsacks (Badanidiyuru et al., 2013 Badanidiyuru ct al. 2018
1.1 The diversity setting introduced by Celis et al. (2019)
Definition 1. A model D is collection of distribulions u with fovile first moments denoled by = E(v).
A K armed bandit problem y = (Vi,..., vK), unknown to the learner, is fixed. At each round, the learner picks an arm A, E [K], where K f1..... K}, and obtains a payoff Y, drawn at random according to VA, conditionally to the choice of A,. This is the only observation made. So far, the setting described is exactly the one of vanilla. K-armed bandits; the distinguishing feature of the bandit model by Celis ct al. (2019) is that the choice of A, is uclually made in two steps, follows. First, distribution 2, over [K] is picked, in some known closed set P of the set MM,+ ([K]) of all probability distributions over [K]. Distributions in the set P satisfy some drawn at random according to 2.- Following game-theoretic terminology, we will call E [K] pure actions arms, and p P mixed actions or distributions.
constraints (specific examples are given below). Then, the arm A, is
We measure performance in terms of expected payoffs. More precisely, denoting by μ = E(v,) the expecta- tion of vy and by μ = (H1,....,,kk ,Kk) their vector, we first note that at round t > 1, by repeated applications of the tower rule,
B At, 2,] MA,, thus B(Yc = Σ kE[K]
Pc. Pkk def (2,11,
11)
thus E[YY) = Eke, 12). (1)
2

---

Published in Transactions on Machine Learning Research (07/2024)
Box A: PROTOCOL OF DIVERSSITY-PRESEEVVIIG STOCHASTIC BANDITS (CELIS ET AL., 2019
Known paramcters
Arms 1, ..., K and model 'D of distributions for the arms
Closed set P P ([]+ of diverse enough probability distributions over the arms
Unknown parameters
Probability distributions y (M1,..., ,K) in D, with expectations μ = (11 (M MK)
For 1 /= 112.....
1. Pick a distribution P, =
E P over the arms
2. Draw at random an arm A, ~ P,
3.
Get and observe a payoff Y; VA, drawn at random according to vA, given A
Aim
Minimize the diversity-preserv expected regret
Rr = T max
E
P 4
peP
We consider expected regret: its distinguishing feature compared to vanilla K armed bandits is that the performance the learner compared to distributions p E P only, not all distributions MM,+ ([KT). More precisely, we define the siiversity--reservvi expected regret as
RT
max (p. p) E PEp
Σ Pe 11)
The diversity-preservingg expected regret is smaller than the vanilla expected regret for K K-armed bandits, which corresponds to comparing to the maximum over p E M1,+((K)) In some cases (see Theorem [ and expected regret may even be bounded.
Corollary D the
The learning protocol and the regret goal are summarized in Box A.
Example 1 (Avoiding polarization). The main example by Celis et al. (2019) consists of imposing that each arm should be played at each round with some minimal probability i e>0, 0, which corresponds to the Seversity-preeevingg set p = {p: Va E [K], Pa >> {}. This constraint makes sense in online advertisement: all offers need to be displayed a significant fraction of the time and get significant chance to be selected. As argued by Celis al. (2019), this wish of diversity would not take place with classical bandit algorithms, which would display almost only the most profitable ad (a phenomenon called polarization). More generally, for each arm a, depending the contract passed, there could be known range [La, Uo] of individual probabilities of display, so that
P p={2: Va E [K], Pa E (La, uel
Example 2 (One-shot version of bandits with knapsacks in the mechanism design). This second example is own Suppose that every pure action is associated with N costs c(') ...., (N) in R, accounting for limited resources or environmental costs like the amount of carbon emissions generated from taking the action; negative costs (e.g., negative carbon emissions) are allowed. When player picks a pure action A, according to the mixed action p = (PI....,Pk) the N expected costs associated with her choice are
Σ [E[K]
Pac
11)
Σ Po RE
In this case, a reasonable objective for the player is maximize her payoff under the constraints that, for all [N], the n-th expected cost of her actions be kept under certain pre-specified level un R. This
3

---

Published in Transactions on Machine Learning Research (07 (274))
amounts to playing with the probability set
K -: Vn E Eaac & Lnn
a=1
Note that the name seliversty-prrserviing was inspired by the example of the previous paragraph and is perhaps less relevant in the present example. The present example is a one-shot version of bandits with
knapsacks (Badanidiyuru et al.) 2013 Badanidiyuru et al. 2018), where the budget constraints must satisfied all rounds but the mechanism design, and not when actual costs are summed over all rounds (though the latter will be approximatively achieved by martingale convergence).
In both examples, the diversity-preservii sets P considered are polylopes in the following sense. Definition 2. A polytope is convex set generated by finitely many extremal points whose set is denoted by Ext(P).
When P is a polytope, we may introduce suboptimality gaps of distributions Ext(P) and decompose the snersity-preservingg regret as stated in distributions, which achieve an expected payoff denoted by M (p, P), i.e.,
2 More precisely, we denote by Opt(z, P) the subset of optimal
M P) = max (p, 11
and Opt(y, P) = argmax u)
pEp
pEP
and define in turn the suboptimality gap 4g of a given distribution p Ext(P), and the global suboptimality gap A of the set Ext(P),
Ap = (x (2 and
A min Ap: E Ext()), 4p Sp>03
(We assume that at least one p E Ext(P) such that Ap > 0, otherwise, all strategics have a mull regret.) regret
Denoting by (T) the number of times a distribution E Ext(P) is played, the of (possibly randomized) strategy only picking distributions P. ExI(P) can then be rewrilten
RT T max((p μ² pEp
Pp 1))
Σ
42 (N))) where N₂(T) Σ
L =2 (2)
DEExt(PN.Optt..))
consider the possibility of contexts in their description of the setting, but then introduce and analyze policies that "function independently for each context". The setting and aim described above and summarized in Box A correspond exactly to the framework of Theorems and 2 of Celis et al.
Actually, Celis et al. (2099
(0099h
1.2 Overview of our results and comparison to Celis et al. (2019)
Celis al. (019)) provide some extensive numerical studies but we only discuss below their theoretical contributions; the latter only dealt with eiversity-preserving sets P given by polytopes.
Regret upper bounds by Celis et al. (2019) : for polytopes. They first suggest using linear-bandit
algorithrs in the diversity- setting, since ① indicates that the expected reward is a linear function of the played distribution P, Of course, with such a reduction, the learner discards a some useful information: the value A, of the arm actually played. This is why Celis et al. (2019) obtained suboptimal regret bounds. More precisely, they use the LinUCB stratogics (lincar upper confidence bound) strategics introduced by Li et al. (2010) and Chu al.] (2011) and further studied by Abbasi-Yadkori et. al.) (2011). With this stralegy, they obtain regret bounds of order K (In T) /A. A second suggestion by Celis et al. (2019) is a constrained version of the e-greedy algorithm by Auer et al. (2002). which does use the knowledge of the arms A, actually played and obtains regret bound of order (In T)/A. However, this constrained e greedy algorithm requires the value of A to tune its exploration rale e, over time; this limitation makes it impractical compared to knowledge-independent algorithms.
For both strategics of Celis et al. (2019). the case of a bounded regret is not covered, while constitutes our main contribution.
4

---

Published in Transactions on Machine Learning Research (07/2024)
Regret upper bounds for polytopes in this article: Section 2 and Appendix A More precisely, we introduce in Section 2 diversitt-preeeevinng UCB strategy, close to the original UCB strategy by [uurr et al. 2002] it maintains indexes on arms and outputs the best distribution in P given these indexes. Additional minor modifications are needed as the strategy cannot ensure, unlike in the classic case, that all arms are first pulled once. For sub-Gaussian models and for diversity-preservinn sels P given by polytopes, this diversity-preeerving UCB strategy enjoys a bounded regret as soon as all optimal distributions for a problem put some positive probability mass on each arm; this condition somehow indicates that each arm is desirable and flags cases where diversity is welcome. Even when the condition is not met, a In T regret is suffered.
Both regret bounds are stated in Theorem 1 whose proof may be found in Appendix A The proof of the In T bound relies mostly on typical optimistic techniques for K-armed bandits, with occasional twists, to provide controls on the numbers of times arms a E [K] are played through information on the numbers of times distributions p E Ext(P) were picked. The proof of the bounded regret follows a completely different logic: we first show that optinal distributions are lypically played al least half of the time, which, entails, because Pmii (2) > 0, that each pure action [K] is played linearly many times. Therefore, all estimates are sharp, and little regret is suffered. The proeeittp setting has it here that all arms may be, and even should be, played linearly many -ndese in the vanilla K ---reedd bandit settings, where suboptimal arms are only played about In T times.
Regarding computational complexity, the algorithm we propose only requires solving one linear program over the constraint set at every round, which is often less expensive than the double maximization over an ellipsoid of LinUCB (sce Li et al. 2010 Chu et al. 2011 and Abbasi-Yadkori al. 2011).
Regret lower bounds in this article: Sections and Appendices
Celis et al. 2019 do not
provide lower bounds. We provide optimality results in Section 3 (with proofs located in Appendices B and C They are stated under the restriction that. the bandit. problem v considered has a unique optimal distribution (z)). Theorem 2 states that in models with upper bound on the expectations (and satisfying two other mild technical conditions), all reasonable strategies (i.c., achieving small diversity- regret, for all bandit. problems) must suffer a In T when p" (2²) puts no probability mass on al least one arm. Together with the upper bounds results discussed so far, we therefore obtain in Corollary [a characterization of the bounded versus In T regrets for these models with no upper bound on the expectations.
The case of models with bounded expectations is more delicate and we illustrate in Proposition with Bernoulli bandit problems, why stating such characterization is challenging.
The proof techniques rely to a large extent on classic proof schemes for bandit lower bounds Lai & Robbins, Graves & Lai. 1997 Garivier et al. 2019). The main difference raised by our setting is described in
1985
Remark 1 eventually, arms A, are played, so that the Kullback-Leibler information gain should be quantified in terms of the A, and is larger than if it was quantified in terms of the distributions p, used; yet, the small- regret constraints on the strategies are in terms of the distributions 2, used. This leads to some specific constrained minimum on lim inf Ry/ In T, à la Graves & Tai 1997), which is to be studied.
Regret upper bounds for P given, e.g., by a ball: Section and Appendix D
We study a case set P is not given by polytope but whose set of extremal points is continuous
where the and curved. For simplicity, we consider Euclidean balls in the probability simplex. We obtain regret bound of order In² T when such ball P lies in the relative interior of the simplex. The striking feature of this bound is that a linear-bandit algorithm discarding the arms At actually played would pay regret of order at least vT, showing again a large gap between knowledge-informed algorithms and generic linear-bandit bounds when all actions are desirable.
1.3 Literature review
The many existing notions of algorithmic fairness have lead to different fair bandit problems formulations see, e.g, the contributions by Wang et al. (2021), Liu et al. (2017), Barman et al.) (2023), which are unrelated to the setting we study. We focus our literature review below on comparable work.
5

---

Published in Transactions on Machine Learning Research (07/2024)
We mention in passing the first version of the present article: Hadiji et al. (2020).
Diversity, Chen et al. (2020) study a particular case of the diversity-preeervii setting, which essentially corresponds to Example While their framework is the same ours, they only study distribution-free
bounds, at best of order v KT. Their algorithm applies to rewards generated in an adversarial manner.
In the domain of stochastic bandits, Patil et al. (2021) and Claure et al. (2020) design bandit algorithms ensuring that the proportion of times each action is selected is lower bounded, i.c., with our notation, that N,(T)/T > Ua at every round. This is similar in spirit to the special case of our setting presented in Example | However, the hard constraint on the number of pulls poses algorithmic design issues different from ours, which they solve only in the special case of lower bounds on every arm pull. Our setting is more flexible yet enforces similar guarantees while bypassing these issues.
After a preprint version (Hadiji et al., 2020) of the present article was published online, Liu et al. (0222) building on Li et al. (2020), considered a combinatorial bandit problem with constraints on the distribution over actions selected by the player. While they formally allow the player to play outside the constraint set, they measure regret with respect to the best constrained distribution, and their algorithm never plays outside the constraint set, making their sctting essentially identical to ours. They analyze a natural extension of diversity-preservinn UCB to the combinatorial selting, providing logarithmic regret bounds similar to our Theorem D and discuss specific cases in which constant regret can be achieved, e.g., when all distributions put some given positive probability on all actions. Compared to their results, we provide characterization of when constant regret happens-thi goes way beyond the case of a positive lower bound on the components of the distributions. Anecdotally, our upper bound on the constant regret, when specified to Example D is tighter than theirs. Indeed, the main term in our bound of order K/(A Pinin (im)(()) up to log terms, whereas they get a bound of order 1/(a (Pinn (2)²), which is always larger since Pinin (2) < 1/K.
Note that all these articles refer to "fairness", although we use the term seiversity-preserviin in this case.
Bounded regret in structured bandits. A recent line of work (Hao et al.
2020
Tirinzoni et al. 2020
Jun & Zhang 2020), pioncered by Bubeck et al. (203)) and Lattimore & Munos 2014)) studies the possibility of achieving bounded regret in structured bandits. There, the learner knows a priori a structural property of the relationship between the payoffs of different actions available. Certain types of structures give rise to the "automatic exploration" phenomenon: despite the bandit feedback, playing the optimal arm (and knowing the structure) yields enough information to diffcrentiate it from the suboptimal arms without playing them; this opens up the possibility of obtaining bounded regret.
These results are deeply connected to the lower bound of Graves & Lai (1997). which often provides the tight constant in front of the logarithmic asymptotic rate of regret in structured bandits, matched in wide generality by Combes el al. (2017); see also the equivalent formulation for linear bandits with Gaussian noise in Lattimore & Szepesvári (017)) Constant regret can only occur in problems for which that lower bound is O.
As in most bandit settings, these works all assume that the learner has deterministic control over the arms A, uvllt picked. The diversity- framework contrasts this aspect, by preventing the learner from picking A, and introducing some additional random draw of A, based on the distribution p, picked. The relationship between the policy and the information acquired is thus altered in a central way. On a technical level, standard bandit algorithms may decrease mechanically the radius of the confidence bounds, whereas the extra randomness between 2, and Ar only allows for some probabilistic control in the diversity- preserving setting. In particular, the analysis techniques of Lattimore & Munos 2014) and Tirinzoni et al.) 22020), based on confidence bounds, need to be sharply refined to be used in the diversity-preserviin setting,
Also, to the best of our knowledge, the results of Section 4 provide the first example of a structured bandit problem with an action set (the diversity-preservvi set P) with a continuous set of remal points and for which automatic exploration occurs and brings the rate of regret from vT down to In2T.
Note that automatic exploration also happens organically in Degenne et al. 20188) where the authors assume directly that the learner may observe the payoff of extra arm.
6

---

Published in Transactions on Machine Learning Research (07/2024)
Mediator feedback. The with mediator feedback, on the one hand, and with regret minimization with expert advice, on the other hand. In all three settings, arms are not directly pulled but a distribution (possibly non-stationary) is placed in the middle.
setting of this article has notable connections with bandits
Inspired by policy optimization reinforcement learning, Metelli et al. (2021) introduce the problem of bandits with mediator feedback, which is more general version of the prreeevipp problem we consider. They define bandits with mediator feedback as any bandit problem in which the player observes intermediate (possibly random) feedback o that determines the reward. (Precisely, the reward Y, is independent of the action As conditionally on 0t.) The authors analyze, in particular, the case of policy optimization within a class of policies parameterized by some 0 E θ; the goal of the learner is to select policies (parameters) that yield high returns. In this example, the player observes the entire sequence of rewards and actions throughout the episode, which more informative than the sole cumulative gain of the policy. Our setting is more specific but our analysis is also more precise. The results by Metelli et al. (2021) can indeed be applied to our setting, by considering Markov Decision Process with single state and by identifying the policies directly with distributions over the action set. Based this reduction, their analysis does not make full use of the simple information structure available, which results both in suboptimal lower and upper bounds. More precisely, the regret bound of their algorithm becomes infinite as soon as P is fully not contained in the relative interior of the simplex, while our regret bounds are at worst of order In T. (In their notation, p corresponds to 0, and v(0) = oo.) On the lower bound side, they only provide a worst-case result, by exhibiting hard instances on which algorithms incur a minimal regret, whereas we characterize the optimal rates on every instance.
Another article worth to be mentioned is by Eldowa et al. (2023), in the context of regret minimization with expert advice: indeed, the finite policy set θ that they consider in K'armed bandits could correspond to the set Ext(P), which makes the setting studied here a special case of theirs with i.id. stochastic losses, However, to the best of our reading, we found no trace of constant regret and only bounds scaling with vT. This due to the adversarial approach followed; their angle is to improve the regret bounds terms of their dependencies on other quantities than T (e.g., number of policies, arms, etc.), which they do through the information--heorctii quantitics introduced.
Extensions to Markov decision processes. Occurrences of constant regret have also recently appeared in the reinforcement learning literature, in episodic settings more or less related to ours. For linear Markov Decision Processes, Zhang et al. (004)) derive algorithms that achieve constant regret with high probability, but with polentially large expected regret, and Papini et al. (2021) extend the work of Hao et al. (2020). discussed above, providing a necessary and sufficient for constant regret. |Vera al. (2021) introduce an online version of knapsack problems that accounts for the computational tractability of the benchmark strategies, and design constant regret algorithms against these benchmarks. Wagenmaker & Foster (2023) tackle the
general and difficult question of characterizing the non-asymptotic instance-dependent optimal regret in reinforcement learning (contrasting with the asymptotic nature of the lower bounds following approaches à la Graves & Lai 1997). Therein, constant-regret instances play a special role, since the constants, which are typically -oddn terms, form the dominant term in the regret.
Note the difference with the setting of Metelli et al. (2021) discussed earlier: in their formulation of policy optimization, the action set corresponds to the set of parameters, and not to the action set of the MDP in the references right above.
2 A simple peeesstty-rrseevving UCB strategy
Definition 3. A distribution a over R with expectation = E[Vv is o2-sub--a Gaussian if
f de(o)) & 102
We assume that the model D composed of oussub-Gaussian distributions, where the parameter o² is known. A lemma by Hoeffding (1633) indicates that the model Dja,)) of all probability measures supported
7

---

Published in Transactions on Machine Learning Research (07 (022))
Box B: DIVERSITY-PRESERVING UCB FOR POLYTOPES
Inputs: sub-Gaussian parameter o² for distributions in D; polytope P
Initialization: pick some no taken by some distribution in D, and let ([)) = (uo: ..... mo)
For rounds = 1,2.....
1. Select (ties broken arbitrarily) a distribution
P. argmax (p, Uit - 1))
pEEx(()
2. Play the pure action A, ~ P,
3. Get and observe the reward Y2 VA.
4. Compute the empirical averages
if (1) 0,
(t)
where No(t) Σ (AA_=a)
Σ Y.
if Na(() > 1,
(1) am1
5. Compute the upper confidence bound vector Utt) (Ui(t). ... Uk(t)) according to
8o Int
Va E [K],
Ua(t) = pa(t) + max (t),
on a compact interval [a, b] satisfies this assumption, with o² (b a)3 Of course, the model D of all Gaussian distributions with variance smaller than some prescribed o² is also suitable.
We also impose for now that is a polytope. See Section 4 for
where P not a polytope.
We then introduce a UCB strategy in Box B, which maintains indexes U. (t) directly and separately on the arms E [K] as the classic UCB strategy. The strategy neither uses indexes for distributions in Ext(P), nor
resorts to some global linear-bandit estimation as in Abbasi- Yadkori et al. 20111). The indexes are initialized
at some value uo possibly taken by some distribution in D, and are of the form, for t > and a E [K],
80 Int Ua(t) = (t) max (Na
where (!) counts the number of times arm was pulled till round t included, and where (t)( is the empirical payoff achieved over those rounds when
played. Of course, it may be that Na 0, the learner has no direct control on which arm is played-the learner cannot ensure that arm be picked even once. This is why we use max (N), in the confidence bonus, and also, set (ta() up in the case (t)+)= 0.
We prove (in Appendix Ap the regret bounds stated below Theorem Theorem 1. Let P be polytope. Consider lub-Gaussian model D with parameter o², known and used by the diversity diversity-preservinn preserving UCB strategy of Box B. The regret of the latter satisfies
Yu in D, 11
Rr < Cy In +
where Cr and Co are quantitics depending on v and whose general closed-form expressions may be read in 12 In addition, for problems y. in such that
Pinin (v) def min min Pa > 0, the regret is even bounded: Pe
lim sup Rr < t00;
T-x
similarly, a closedl-fomm finile upper bound may be read in 19
Some comments are in order, first on the cases of interest (and those that are mere sanity checks), and second, the proof techniques.

---

Published in Transactions on Machine Learning Research (07/2024)
Cases of interest in Theorem 1 The In T bound in Theorem could also be achieved (with the same computational complexity) by a UCB strategy taking the distributions in Exi(P) arms. The main achievement in Theorem is therefore the case of bounded regret. The condition Pinin (1) > 0 corresponds to cascs where diversity is indeed desirable, and this is when bounded regret is achieved.
The diversity-preserving UCB strategy of Box B coincides with the classic UCB strategy by AAer et al. (2002) in the case where P is the entire simplex of probability distributions. However, the general In T bound achieved in Theorem [ with closed-form expression
12 does not coincide with the classic UCB bound of Auer et al. (2002) in this case. This is due to some general but loose analysis followed in the proof of Theorem 1 We recall, once again, that the In T part of Theorem only forms some sanity check, the main achievement being the bounded-regret case.
Intuition of the proof and proof techniques. The proof of Theorem | is provided in Appendix A actu- ally, two separate proofs with two different logies are provided: one for the general In T rate in Appendix A.1 one for the case of bounded regret in Appendix A.2
The proof of the In T bound was obtained by an adaptation of the classic UCB proofs. for vanilla K K-armed bandits (see Auer et al. 2002) or for linear bandits (sce, c.g., Abbasi- Yadkori et al.) 2011 and Lattimore & Szepesvári 2017). In particular, the proof shows, in view of (2), that suboptimal distributions e Ext(P) are unlikely to be played more than I times. We highlight in Appendix A the (three specific) technical modifications with respect to these classic proofs: they all basically amount to controlling, with high probability, the (a) based on the numbers of times Np(1) distributions p are played.
The proof for the case of bounded regret follows a completely different logic (the challenge to be overcome here was to get and write down this alternative logic). We first show that optimal distributions are typically played at least half of the time. This entails, because Pinin (V) > 0, that each pure action [K] is played lincarly many times. Therefore, all estimates the rationale for the bounded regret: there is less competition than in the classic case bet ween exploration and exploitation exploitation involves exploration "by design" in the case where (2) > 0. This rationale is different from the one at hand in structured bandits (sce Section 1.3 for the reforences): therein, arms played also provide some information on arms not played thanks to the underlying structure.
sharp, and little regret is suffered. This proof thus provides
3 Lower bounds / Optimality in the case of polytopes
In this section, we discuss the situations where the In T' rate stated in Theorem D is unavoidable: does this happen when optimal distributions put probability mass at least one arm? To answer the question, we first provide in Section 3.1 (more preciscly, in Thcorem 2 In T lower bound on the regret against all bandit problems with a unique optimal arm not putting any probability mass on some arm, provided that the model is mean-unbounded in the sense of Definition 4 below.
In Section 3.2, we the combine the lower bound of Theorem P and the upper bound of Theorem D to show that for sub-Gaussian mean-unbounded models, either the regret may be bounded or it must grow at an optimal In T rate. The case of mean-bounded models is more delicate and Proposition T illustrates the intrinsic difficulties in characterizing the rates for these models.
Before we proceed, we state our key distinction between mean- mean-bounded bounded and mean mean- unbounded models. Definition 4. A model D is said to be mean-bounded if there enists M E R such that the expectation μ = Flv) of any v E satisfies μ & M. Otherwise, the model D is said mean-unbounded: for all M E R, there exists D such that μ = E(v) > M.
Example 3. The rewards in online advertisement (see Exomple i are naturally bounded. Admittedly, most real applications are associated with mean-bounded models; mean-unbounded models (that are also sub-Gaussian) are less common. We could think of the model with distributions of the form: some mean plus some fized, possibly wide-spread, sub- Lub-Goussian centered distribution. It would model, e.g., in finance applications, returns of unknowm levels but with known common shape. The prerrstttppreeruing constraint could come from the necessity of putting all inwestments at a given round in a single asset while considering safe allocation policies that pul, in expectation, a fraction of the capital in all assets.
9

---

Published in Transactions on Machine Learning Research (07 2224)
3.1 General regret lower bound
We restrict our allention to models salisfying the following technical assumption, which, we discuss below; is mild and is satisfied, in particular, for convex models D.
Assumption 1. For all distributions E D, for all real numbers 3. if there crists Ç E D with expectation E(() > x, then there erists Ç' E 'D such that E(ç') > x and KL(,,C") to0.
This assumption of course holds for models where all pairs of distributions exhibit finite Kullback-Leibler divergences (for instance, models given by exponential families). It. also hold for convex models; indeed, for À E (0, 1) small enough, <' = (1 - X) + Av belongs to D and still has an expectation larger than x, while the density of y with respect ç' upper bounded by 1/2, thus (L( & 1/A < too. Note that convex combinations of sub-Gaussian distributions with parameter o² are also o--sus--S----- Sub-Gauss,nn, by convexity of the exponential function.
A somewhat typical restriction (also considered, e.g., by Lattimore & Szepesvári 2017 Combes et al 2017
will be issued concerning the bandit problems possibly considered in the lower bounds: they should have a single optimal distribution in P, which we denote by 22()), i.c.,
Opt(2. P) = (2))
Of course, when P is a polytope, that single optimal distribution (2(2) belongs to Ext(P).
The next, extremely mild, assumption is on arms: we require that there are no unnecessary arms given P, i.e, that each arm in [K] may be pulled for at least one distribution in P, or equivalently if P is a polytope, in Ext(P).
Assumption 2 (no unnecessary arms given P). For all a E [K], there exists p e P such that Pa > 0.
Finally, given the regret upper bounds exhibited in Theorem it is natural to restrict our attention to so-called uniformly fast convergent [UFC] strategies.
Definition 5 (UFC strategics). A strategy is uniformly fast convergent [UFC] over the model D given the pnbersity-preserving set 'P if for all bandil problems y in D, its diversily-preesevity regret salisfies Rr = o(Ta) for all 0.
The main result of this section is the following theorem, proved in Appendix B We actually provide therein general lower bound holding for large classes of models mean-bounded and mean-unbounded ones.
Theorem 2. Consider mean-unhounded model DD obiding by Assumption 1 and diversity diversity-preservvin preserving polytope P such that there are unnecessary arms given P (see Definition D with single optimal distribution in P denoted by p" (2), if
For all bandät problems v in
3a E [K] (T(()) = =0,
lim inf RT
then there erists L, > O such that for all strategies UFC over DD given P, A closed-form expression of a suitable L, is given by Lemma in Appendir B.1
> Ly.
7-000 In T
3.2 Characterization, or lack of characterization, of the optimal regret rates
Theorems and 2 entail the following characterization of regret rates in the case of sub-Gaussian mcan- unbounded models.
polytope P such that there are no unnecessary arms given P (see Definition 2), for a mean-unbounded model Dy² compossed of ou- Gaussian distributioos and abiding by Assumption for all bandit problems y in Dz= with single optimal distribution p² (z) in P,
Corollary 1. For a
f e [K] : (z) > 0, the siversstyypreserving UCB strategy ensures that the regret Ry is bounded; B [ ()) 0, then all stralegies UFC over D given P suffer logarithmic regret.
10

---

Published in Transactions on Machine Learning Research (07 (204))
On the contrary, the case of mean-bounded models is delicate. The (counter-Jexampll below deals with the model of Bernoulli distributions. The characterization of rates obtained looks idiosyneratic: it seems intrinsically difficult to provide general characterization of bounded regret by the expected payoffs in the case of mean-bounded models.
Proposition 1. For the model B of all Bernoulli distribulions, K = 3 arms, and some å E (0, (0, consider the tioversity-pesserving segment Ps generated by (())== (0, 1/2, 1/2) and (3)) = (6. 0, 1 5). Let the bandit problem be y (Ber(0), Ber(1/2), Ber(O)). The setting introduced satisfies all assumptions of Theorem except the mean-umboundedness of the model, and p is the unique optimol distribution for Yet,
RT
if > 1/4, there evists Ly > 0 such that for all strategies UFC over B given Ps, lim inf T-500 In
> byj
ifō < 1/4, variant of the diversity-preserving UCB obtains lim Rr < t00.
T-
4 Example of pressevy- set P given by a ball
In this section, we discuss a specific example of a diversity preserving set P that is curved with smooth boundary 0P. Our aim here is to illustrate that in the non-polytope case, there can a wider variety of rales for the growth regret-beyond the In T rate and bounded-regrel cases discussed earlier in the case of polytopes. Our conjecture (given the proof technique of Theorem below) is that the In T rate for the regret exhibited for the specific example also holds more gencrally for convex sets with local quadratic curvalures. We leave the general study of the non-polytope case to future research.
Specific example studied. Our running example will be the intersection of the probability simplex with Euclidean ball centered at the uniform distribution,
P,
p:
Σ (Pa 1/K)2 Er² aE[K]
for small enough. This example differs from the case of polytopes in that the set potential best actions becomes infinite (continuous): it is still given by extremal points, which are now given by the boundary 0p,. In particular, what corresponds to the minimal gap A is now equal to O.
In standard lincar bandits with actions sets given by such convex sets P,,, the logarithmic regret bounds, which depend on the inverse gap, become vacuous and the best problem-dependent rates degrade to vT. ef. AAbbssi- Yadkoii et al. (2011) and Lattimore & Szepesvári 2020)) Banerjee et al. (2023) Theorem 3.3) proves that reaching sub-vT VT' regret on ellipsoids is essentially impossible by showing that any algorithm with worst-case regret smaller than vT must explore all directions a proportion vT of the time.
In the diversity setting, we state below that version of the diversity- UCB strategy enjoys regret bound of order In instead of the typical vT bound that linear bandit algorithm not taking into account the arms A, played would incur.
Statement of the strategy. The formulation of this to the Box B strategy, except that the set of extremal points is now given by the boundary 0P, of P, and is infinite; we therefore specifically pick, in Step 1 of Box B:
UCB strategy is almost identical
P, argmax (p, U(t ((t--))).
pEap,
Note in passing that computing 2, comes at no computational cost: a simple closed-form expression follows from Lemma of Appendix D
If the radius r small enough, then P, is contained in the relative interior of the probability simplex, guaranteeing that every pure action [K] gets some minimal positive chance to be played at any round.
11

---

Published in Transactions on Machine Learning Research (07 (274))
More precisely, the condition reads
Plim
E K K-1
for which
min min Pa Plim
(5)
PEP, aE(K]
The value Piim is the distance from the uniform distribution vector to the uniform distribution all but one pure actions. The following asymptotic result is proved in Appendix D
Theorem 3. Assume that r < Tim Consider sub- Gaussian model D with parameter o², known and used by the is al most squnred-lggarthhmic
UCB strategy 4 on Pr. For any bandit problem in D, the regret of this strategy
RT T < t00.
lim sup T-000
While this result should generalize to other strongly convex diversity sets, we stick the repre- sentative example of Pr, which simplifies the exposition of the proof. In particular, of Pr to provide explicit formulas for the optimal distribution(s) and optimal payoff (see Lemmas 6 and in Appendix D). Similarly, we focused on the asymptotic In T rate with no attempt to provide a closed-form regret bound with explicit constants. These constants would depend on and on the curvature of P in a complex way, and we chose ignore this aspect here.
the simple shape
5 Some experiments on synthetic data
In this final section, we perform some (simple and preliminary) experiments that merely illustrate the dual behavior of the regret: either bounded or growing at a In T rate. We believe that a more extensive empirical comparison would be interesting but would be out of the scope we targeted for this article.
Experimental setting. We consider K arms and the model D of Bernoulli distributions. The preeeeiny set P is the triangle generated by
p = (0, 0.2, 0.8),
p (0.6, 0.2, 0.2), and p (0, 0.8, 0.2).
We consider the bandit problems v. with expectations
H = a,
where a 0.1}.
Lemma 1. For 0.1,, the unique optimal distributton is 222) which puts posilive probabilily mass all actions. For a = 0.1, the unique optimal distribution is (()) which does not put any probability mass on action 1.
Proof. First, the distribution (3)) is always dominated either by (1)) by p(2), and is therefore never optimal distribution: for all a E (011, 0.1},
(p(3) (1)) pa) = 0.6 (13 = 0.6o) < 0;
13 22))
(2) ( = 0.6 0.6 (-(1/22a)))+1/ - 0.6a < 0.
We now compare the mixed actions (22) and 22);
( - 22)) --6(-----=2) a)) 1.2x.
The sign of thus determines which of (1) (2)) is the optimal distribution.
12

---

Published in Transactions on Machine Learning Research (07 (07/2024)
p" 10= 0.8]
DivP-UCB L1-OFUL
10a
10a
10
10a
10a
10a
10a
10*
10a
=0.6
DivP-UCB L1-OFUL
10a
10a
10a
10"
10
102
10a
10*
10
Figure 1: Estimated expected cumulative regret over time, in the case a = -0.1 [top figure, bounded regret] and a = 0.1 [bottom figure, In growth], for the two algorithms considered. Solid lines report empirical means while shaded areas correspond to standard errors of the series defining the empirical means.
Numerical experiments. We ran the diversity-prreserving UCB algorithm (see Box B, abbreviated DivP- UCB below), as well as Algorithm 2 (Constrained-L,-OFUUL. abbreviated L1-OFUL below) of Celis et al. 2019) We did so on each of the two problems 2,, over T = 100,000 time steps, for N = 100 runs. The expected regret suffered by each algorithm is estimated by the empirical averages pseudo-regret regrets observed the N runs:
T Σ (2a) p,,a, H.), [=1
Rr(wa)
Rr(, i), where Rr i)
and where we denoted by P, p, i) the mixed action chosen at round t, during the th run, for problem K.. Figures report the estimates Ry (2.) obtained (solid lines); shaded areas correspond to #2 standard errors of the series (2,,,,)) i) used in the definition of the Rr(,,) as empirical averages.
As expected by combining Theorem I and Lemma 1 we observe a logarithmic regret growth when a = -0.1 and bounded regret when a = 0.1 for DivP-UCB, whereas L1-OFUL incurs logarithmic regret in both cases. It also turns out that on this specific example, DivP-UCB performs better than L1-OFUL. (Note also that L1-OFUL is computationally more costly than DivP-UCII since it needs to solve 2K linear programs every time step, instead of single one for DivP-UCB.)
13

---

Published in Transactions on Machine Learning Research (07 (07/2024)
6 Conclusion and limitations
We revisited the piversit variant of K armed bandits introduced by Celis al. (2019): we considered the same framework as introduced therein (while extending it to mab-Gaussian models), with diversity-preserviig sets P given by polytopes; but we stated and discussed a more satisfactory UCB-based strategy, controlling the regret by a In rate in general and even achieving bounded regret in the case where all arms gel a positive probability mass by all optimal distributions i.e., when diversity is desirable. We showed that this dual behavior (and the associated condition) were optimal for unbounnd models, while pointing out, through Bernoulli counter-cxample, the intrinsic difficulties in charactcrizing the rates in the case of mean-bounded models. The latter issue is the first question left for future research. We then also discussed specific example of grows at worst at In ' rate. Here well, building a general theory for curved P is left for future rescarch. Finally, we proposed some simple and preliminary numerical simulations, that could be extended.
set P not given by polytope, where the regret sets
Acknowledgments
The work of Sébastien Gerchinovitz and Jean-Michel Loubes has benefitted from the AT Interdisciplinary Institute ANITI, which is funded by the French "Investing for the Future - PIA3" program under the Grant agreement ANR-19-PI3A-0001. Sébastien Gerchinovitz gratefully acknowledges the support of the DEEL project (https://www deel. deel.ai/). ai/).
References
Yasin Ubbasii-adkonn David Pål, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Newral Information Processing Systems (NeurIPS '11), volume 24, pp.
2011.
Peter Auer, Nicolò Cesa Bianci,, and Paul Fischer. Finite-time analysis the multiarmed bandit problem. Machine Learning, 47(2-3):235-256 2002.
Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. In IEEE 54th Annual Symposium on Foundations of Computer Science (FOCS113) pp. 207-216, 2013. Latest extended version available at arXiv:1305.2545: dated September 2017.
Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with global convex con- straints and objective. Journal of the ACM, 65(3))1-55, 2018.
Debangshu Banerjee, Avishek Ghosh, Sayak Ray Chowdhury, and Aditya Gopalan. Exploration in linear bandits with rich action sets and its implications for inference. In International Conference on Artificial Intelligence and Statistics (AIStats 2003), volume 206 of PMLR, pp. 8233 8262, 2023.
Siddharth Barman, Arindam Khan, Arnab Maiti, and Ayush Sawarni. Fairness and welfare quantification for regret in multi-armed bandits. In Proceedings of the 37th AAAT Conference Artificial Intelligence (223) volume 37(6), pp. 6762 6762-6769, 2023.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic
Theory of Independence. Oxford University Press, 2013.
Sébastion Bubeck, Vianncy Perchet, and Philippe Rigollet. Bounded regret in stochastic multi-armed bandits. In Proceedings of the 26th Arnuul Conference on Learwing Theory (COLT'2013) volume 30 of PMLR, pp. 122-134. 2013.
L. Elisa Colis, Sayash Kapoor, Farnood Salchi, and Nishocth Vishnoi. Controlling polarization in personaliza- tion: An algorithmic framework. In Proceedings of the 2nd Annual Conference on Fairness, Accountability, and Transporency (11). pp. 169. Association for Computing Machinery, 2019.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Yifang Chen, Alex Cuellar, Haipeng Luo, Jignesh Modi, Heramb Nemlekar, and Stefanos Nikolaidis. Fair contextual multi-armed bandits: Theory and experiments. In Proceedings of the 36th Conference Uncertainty in Artificial Intelligence (UA12200), volume 124 of PMLR, pp. 181-190, 2020.
Yuan Shih Chow and Henry Teicher. Probability Theory. Springer, 1988.
Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. Tn Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AIStats 2001),, volume 15 of PMLR, pp. 208- 214, 2011.
Houston Claure, Yifang Chen, Jignesh Modi, Malte Jung, and Stefanos Nikolaidis. Multi-armed bandits with fairness constraints for distributing resources to human Leammales, 2020. Preprint, arXiv:1907 PXiv:1907.00315 00013..
Richard Combes, Stefan Magureamu, and Alexandre Proutiere. Minimal exploration in structured stochastic bandits. In Advances in Neural Information Processing Systems (NeurIPS volume 30, pp. 1763-1771. 2017.
Rémy Degenne, Evrard Garcelon, and Vianney Perchet. Bandits with side observations: Bounded vs. loga- rithmic regret. In Proceedings of the 94th Conference Uncertaindy in Artificial Intelligence (UAT2001)) pp. 467-476, 2018.
Joseph L. Doob. Stochastic Processes. John Wiley & Sons, 1953.
Khaled Eldowa, Nicolò Cesa-Bianchi Alberto Maria Metelli, and Marcello Restelli. iiformatiionthhorccie regret bounds for bandits with fixed expert advice. In Proceedings of the 2023 IEEE Information Theory Workshop (TTW2,03), pp. 30-35, 2023.
Aurélien Garivier Pierre Ménard, and Gilles Stoltz. Explore first, exploit next: The true shape of regret in bandit problems. Mathematics of Operations Research, 44(2):377-399, 2019.
Todd L. Graves and Tze Leung Lai. Asymptotically efficient adaptive choice of control laws controlled Markov chains. SIAM Journal on Control and Optimization, 35(3):715 743, 1997.
Hédi Hadiji, Sébastien Gerchinovitz, Jean-Michel Loubes, and Gilles Stoltz. Diversity-preserving k-armed bandits, revisited, 2020. Preprint, arXiv:2010.0187411.
Botao Hao, Tor Lattimore, and Csaba Szepesvári. Adaptive exploration in linear contextual bandit. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AIStats 2020), volume 108 of PMLR, pp.
2020.
Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13 30, 1963.
Wen Huang, Kevin Labille, Xintao Wu, Dongwon Lee, and Neil Heffernan. Fairness-aware Handit-based rocommendation. In Proceedings of the 2021 IEFE International Conference on Big Data (IEEE Big Dula (ola))021), pp. 1273 1278. IEEE, 2021.
Kwang-Sung Jun and Chichong Zhang. Crush optimism with possimism: Structured bandits beyond asymp- totic optimality. In Advances in Neural Information Processing Systems (NeurIPS'208)) 2020), volume 33, pp. 6366-6376, 2020.
Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4-22 22, 1985.
Tor Lattimore and Rémi Munos. Bounded regret for finite-armed structured bandits. In Advances in Neural Information Processing Systems (NewrIPS') 14), volume 27, pp. 550 558, 2014.
Tor Lattimore and Csaba Szepesvári. The end of optimism? An asymptotic analysis of finitc-armed lin- car bandits. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (ATStats 2017), volume 54 of PML.R, pp. 728 728-737, 2017.
15

---

Published in Transactions on Machine Learning Research (07/2024)
Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.
Fengjiao Li, Jia Liu, and Bo Ji. Combinatorial sleeping bandits with fairness constraints. IEEE Transactions on Network Science and Engineering, 7(3):1799-1S8 1813, 2020.
Lihong Li, Wei Chu, John Langford, and Robert Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web (WWW'10), pp. 661-670, 2010.
Yunqi Li, Yingqiang Ge, and Yongfeng Zhang. Tutorial on fairness of machine learning in recommender systems. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Relrieval (SIGIR 2021), pp. 2654 2657, 2021.
Qingsong Liu, Weihang Xu, Siwei Wang, and Zhixuan Fang. Combinatorial bandits with linear constraints: Beyond knapsacks and fairness. In Advances in Neural Informalion Processing Systems (NeurIPS'202) volume 35, pp.
2022.
Yang Liu, Goran Radanovic, Christos Dimitrakakis, Debmalya Mandal, and David C. Parkes. Calibrated fairness in bandits. In Proceedings of the 4th Workshop Fairness, Accountability, and Transparency in Machine Learning (Fal/ML 2017), 2017.
Alberto Maria Metelli, Matteo Papini, Pierluca D'Oro, and Marcello Restelli. Policy optimization as online learning with mediator feedback. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (A2A))022), volume 35())), pp. 6769, 2021.
Mathieu Molina, Nicolas Gast, Patrick Loiseau, and Vianney Perchet. Trading-off price for data quality to achieve fair online allocation. In Advances in Neural Information Processing Systems (NeurIPS-2023) volume 36, pp. 40139, 2023.
Matteo Papini, Andrea Tirinzoni, Aldo Pacchiano, Marcello Restelli, Alessandro Lazaric, and Matteo Pirotta. Reinforcement learning in linear MDPs: Constant regret and representation selection. In Advances in Newral Information Processing Systems (NeurIPS'2021 volume 34, pp. 16371-16383, 2021.
Vishakha Patil, Ganesh Ghalme, Vineet Nair, and Y. Narahari. Achieving fairness in the stochastic multi- armed bandit problem. Journal of Machine Learning Research, 22(174):1 31, 2021.
Nícollas Silva, Heitor Werneck, Thiago Silva, Adriano CM Pereira, and Leonardo Rocha. Multi-armed bandits in recommendation systems: A survey of the state-of-the-art and future directions. Expert Systems with Ayplicalions, 197:1 196:11 2022.
Andrea Tirinzoni, Alessandro Lazaric, and Marcello Restelli. A novel confidence-bass based algorithm for struc- tured bandits. In Proceedings of the 23rd International Conference on Artificial Intelligence and Stalistics (ATStats 22T2)), volume 108 of PMLR, pp. 3175-3185, 2020.
Alberto Vera, Siddhartha Banerjee, and Itai Gurvich. Online allocation and pricing: Constant regret via Bellman incqualitics. Operations Research, 69(3):821-840, 2021.
Andrew J. Wagenmaker and Dylan J. Foster. Instance-opiimality in interactive decision making: Toward a non-asymptotic theory. In Proceedings of the 36th Annual Conference on Learning Theory (223))2023), volume 195 PMLR, pp. 1322-1472 2023.
Lequn Wang, Yiwei Bai, Wen Sun, and Thorsten Joachims. Fairness of exposure in stochastic bandits. In Proceedings of the 38th International Conference on Machine Learning (ICML (2021), volume 139, pp. 10686 10696. PMLR, 2021.
Weitong Zhang, Zhiyuan Fan, Jiafan He, and Quanquan Gu. Settling constant regrets in linear Markov decision processes, 2024. Preprint, arX1v:2404.10455.
16

---

Published in Transactions on Machine Learning Research (07/2024)
A Proof of Theorem 1 Analysis of the diversity-preserving UCB policy
The In bound of Theorem is proved in Appendix A.1 by showing, in view of (2) that suboptimal distributions Ext(P) are unlikely to be played more than In times. The analysis mimics and adapts the proof scheme corresponding to UCB run on Ext(P), with three new ingredients specifieally underlined.
The proof of the constant regret bound of Theorem 1 may be found in Appendix [A.2 and follows a completely different logic. We first show that optimal distributions are typically played at least half of the time. This entails, because Pinin (2) > 0, that each pure action [K] is played linearly many times. Therefore, all estimates are sharp, and little regret is suffered.
A.1 Proof of the In T bound in Theorem
We want control the (+)] by In T, however, the favorable events round > hold rather for quantities based how often the pure actions were pulled:
E(t) -00=Yvo Ya E [K], |ma - -aa(t) <
8o Int max ( (N) and Et()) (())=( Va [K],
80 max (Na(t).
A
voa
V
We also introduce the following events, for any p E P, though we will use them only for E Ext(P)((pt((z) P) in the sequel:
K
8 Int Pay max Na))
A
E" (p.
Σ
<
A first new ingredient. consists of the following inequalities, obtained by distinguishing whether = 0 or Pa > 0 and by Jensen's equality for the square root: for all p P, all t < T, and all n > (65Ko?/A?)) In
8vo2 Int Vn
8vo2Int Int Vn
2v 2 Int
Σ Pu aE(K]
Va
VK<A
v max (npa/2, 1}
KE[K]
thus the following inclusion:
n (Na(t) n = n (Na(1) npa/2] C (p, aE(K)
(6)
a:Pa>0 50
Note also the inclusion E'(1) C (())) (), valid for all PEP.
Now, the second new ingredient, consisting of the lemmaa below, is the key to relate the numbers of times (t) suboptimal distribution p Ext(P) is picked to the numbers of draws Na(t) of pure actions [K].
Lemma 2. Fix p E Ext(P), and denote by Pmin>o min(pa a E [K] s.t. Pa 00 its minimal positive component. Then, for all t 1, all (10/ (10/mmiuz)) In T,, and all [K].
( ((N > n (N.(0 < npp//))
< 7
Proof. We only need to show the incquality for E [K] such that Pa O. We note that
N.(1) Σ M/ mot [A==a);
thus, by optional skipping (see Theorem 5.2 of Doob 1953
Chapter III, p. 145, see also Chow & Teicher
1988 Section 5.3), the distribution of N₂ (1) on the event (N)(1) m} is larger than the distribution of
2 immettmes called optional sampling.
17

---

Published in Transactions on Machine Learning Research (07/2024)
random variable Bn,a with binomial distribution of parameters and Pa- In particular,
(((N2(0) > ni n (Nall) < mpp/2))
< (Bnna < nppp22) - P(Bn. - npa < <nppa2) < exp 2( be be/3)
< exp
8(1 11-+11 1/6)
where, for the final incquality, applied Bernstein's inequality (see, c.g., Boucheron et al. Sontion 2.7, Equation 2.10) with variance v Pa [1 Pa), upper bound b = on the range, and deviation = npa/2. 2. Substituting the bound on concludes the proof.
2013 end of
The rest of the analysis is essentially standard. The aim is to control each of the following expectations, for eE Ext(P) \ Opt(v. P) and where up 2 defined later:
T-1
[[)) & u2 +
Σ
P(e = and N₂(1) > nz}
(8)
t=
We first note that for t > 1, for all E Ext(P) \ Opt(u, P),
(p = =p C
C ((t))
(9)
indeed, on E(t) n (",,,)) t), for p E Opt(z, P), by definitions of these sets and of (tt,,
K
80 Int
80 In
(p. (())) = (p, ((t)) +
Paj
max (N,(),,)
< (p. 14 +
Pa
max (1)), 1]
a=1
K
80 Int
< < p
+ App) (p' p) < (p' ((t))
+
Pd
{ (t) U())
max
6=1
while (p.11 = /} requires (p. U()) > (p', U(I)) Lel
65K
10
2 max
A
T
Pmin
In 1+
80
max (Ma aE(K]
uo)2
(10)
the third element in the maximum will turn useful in the application of Lemma beloo.. For each distribution p Ext(P) \ Opt(2, P), the inclusions and then o
entail
(e.11 = (22((2(()) > mp) c E(F) ("
> m2)) S EEt) U (N₂(t) > (N((((((t) < npPa/2}.
[E[K]
Substituting this bound into g
resorting to unions bounds and to Lemma yields
T-1
[[N)(T)) < np + K
P (((())
& En + K +
(2tt
< np + 2K,
(11)
k=n
where we applied Lemma below for each a [K] and with 8 1-, which satisfies the condition required therein given that ng. The proof concluded by resorting to the decomposition (2), to obtain
RT
Σ
4n 2K),
(12)
which is the claimed form C₂ In T In cy. In the derivation of this regret bound, we targeted simplicity and did not try to improve the constants Cg and cy.
Lemma 3 is an essentially standard concentration result for stochastic bandits; the only adaptation therein (the third new ingredient) is handling the case where Nt())= 0.
18

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Lemma 3. Consider model Da² with -ubbb--aussia
Gaussian distributions, and fiz bandit problem in Doz. For L i 1, if the actions AI....,A Ae and rewards YI,... Y, were generated according to the protocol of Box A,
then, for all a E [K], for all 8 > O with 2 2l1(1/6) > (Ma- - u0)
P Ma - (t)l >
20 In(1/0)
E 218
max (Na(t).1)
Proof. Again by optional skipping (see the proof of Lemma 2 by denoting by Pa,n an empirical average of 3/ 1 i...d. random variables with distribution May and by using the convention Pa,0 uo. we have
20 ln(1/o) max { [a(t).1}
20 ln(1/0) max (n, 13
P Nr - ia(1))
P En E {0, (0.1,.....1 ............ |Mm Ma,nl
2. IP
20 In(1/8)
< 0++
Mo Aaam
26 -268;
n
where the case n = 0 was dropped the union bound because |Ma - unl < v 20 In (1/5) by assumption, and where the final inequalities follow from the Shinérr-Cheroff inequality (see, e.g., Lattimore & Szepesvári 2020 Corollary 5.1).
A.2 Proof of the constant regret bound in Theorem
As indicated at the beginning of Appendix A the proof of the constant regret bound of Theorem follows completely different logic. For instance, in Appendix A.1 the sets (l) were instrumental in the proof but we had not controlled their probabilities -hichh which constitutes the core of the analysis here. To do so, we show that optimal distributions are typically played at least half of the time; this is the main contribution of this proof. Then, because Pinin (v) > 0, we know that each pure action a E [K] is played linearly many times, which cannot. happen on the events E'((), where at least one action is only played logarithmically many times. This proof strategy for bounded regret was used in Lattimore the additional technical challenge here that we do not control with certainty the number of pulls of every pure arm because of the randomness in generating the A, from the pt we handle this by carefully applying Bernstein's 's inequality.
Munos (2014). We face
Step 1: Preparation. We fix a threshold to > 8 max (Na u0)2/(80?) such that aE(K]
Vt > to,
2 Pinn (2)
320 Int
Vtlnt
and
At
4 V802 + +22- - > v In (13)
A
For example, with the convention that the In ln the threshold to such that
0o if & 1, the constraints above are satisfied with
o²
3lmh 18432 o²
lo max 2 + 80
In
10
(14)
A 22Pinin (v)
A 2riini (v)²
(Note to reviewers: for the sake of concision, we decided to omit the half-page of calculations that lead to this bound. We could of course add it if deemed necessary.)
By 9 we first note that
Ri RIo
Pa KE[K]
Σ PP
(((((L)
(()))
< Rio +
pa aE[K]
K + E
p(((()))
where the final inequality follows from bound proved in given the first condition to. The key step is the decomposition
E(t) S (N)(t) < t/2)) u (8 (N.(t) (N.(t)+t/2))
where
N.
Σ
Σ PEOpt
1
19

---

Published in Transactions on Machine Learning Research (07 /024)
denotes the number of times optimal distributions are played.
Step 2 (core step): Optimal distributions are typically played at least half of the time. We first deal with the events (1) < 1/2), where / > lo- Le., contrary in the proof of Appendix A.1 we do not only control (t) expectation, but in high probability. To that end, we consider first
(1.(1) < (etl/4):: :o),
where [(1)/4]:00)- = n E(s).
Lr/4]
By classic UCB argunent, for all 8 > 1, by definition of E()),
8a² In S max [N(()).1 Ma & UCB,,a def 2/ max (s),
80 In
on E()), 0 Ua(s) na (s)))) +
thus, for any optimal p- E Optly, P), using also the definition of 2s41 as some empirical best distribution, we have, on E(s).
8 Ins max (s).).
(p' - Ps Pst 1²) -p-,,-u(ss U(s)) (p"
U1s) Poti u(s)) HP U(s) - 11) < Σ Po+
aE[K]
50
{(UCB,,.))E(K))
This inequality and the definition of A as the smallest gap over Ext(P) yield, on {(1/44 : :0);
80² In(s -) 1) max (Na(s-1)) 1),1} #
Σ Σ Pa.a
Le k - 2. 22.
( [t/4] N.))..
== [r/4 aE[K]
As t >/ to > 8 and the sums non-negative terms, proved so far, on & ((t/4)] : o0):
802 In(s 1)
Σ ΣΣ Ps.a
max (Na - 11),13 1 [144] -.()))
(15)
We introduce the martingale
My.t def Σ (pa.a 1(,,=a)) max (Na 1), 1]
8o2 In(s - 1)
2
It turns out that for each E [K], (s increases by if and only if A, a, and otherwise remains unchanged, we have the crude deterministic bound:
802 In(s 1)
1,,==a)
Σ (AA=u)) max (Na(s - 11,11
& 1802In1
max (N(s-1).11 1).1}
< 280 Int
< 2 (1+2/t-1),
n=D mmxx(n.1]
where we used that Na 1) < t determine the range of values for in the sum. The inequality above, together with the relationship and the definition of My,,, entails that
(t) ((t/44) : oo) c ME.: + V80² In t(1 2
2 1) > A(t+1-1t/4 t/2)
S ME, > v 1n
(16)
on tg. As indicated later, the probability of where the second inclusion comes from the final condition the right-hand side is smaller than 1/12.
20

---

2K. Published in Transactions on Machine Learning Research (07 (2742)
Step 3: The remaining events. We now turn to the events (t()) n (N) > t/2], and show that they are unlikely; the intuition is that when (1) is linearly large, because of the condition Piiin (2) > 0, the (t)t) should also be linearly large. More precisely, by the respective definitions of (t) and of Piiiu (2) > 0,
32a Inf A
('()) = U No(1
and Va E [K],
Pa.a >
Σ Mg cOp(ly (m)n( ( Pinin (v).
[K[K]
We introduce the martingales Ma, Na(t)
Σ Ps.a and get, for e to.
(N.(() > S U
U Ma.1 <
32o² Int
2 P Pinin (2) {
t
=
U (Map < -Vlntt. (17)
2
aE[K]
aE[K]
where the final inequality by the conditions side.
13
on lg. We bound below the probability of the right-hand
Step 4: Taking probabilities, via the Hoeffding Azuma inequality.
Collecting the bounds 16
and 17 together, and applying union bounds, we proved so far
T-1
T-1
P (D)
& Σ
Ms(Mas> t=
V802t + Σ (((E(IAA) t=
+
Σ
P(Ma, & - Vtlntt,
t=
dE[K] t=to
and now show that each of these sums finite. We note that P (E(1) in 11 thus, by union bounds,
& 2Ki for l lo, already shown
Σ P (E((t/4]:00) :0)
Σ
Σ
2Ks &
Σ t=o
K
4K
(t/4 1)2
4)2
t= [4/4]
t=
For each a E [K], since the 1,A==) Ps.o form martingale increments with values in a predictable range of total width 1, we have, by the Hoeffding- Booffding-Azaaa incquality, that for all t > and all E >>0 O,
MMM, < - = -P Na(t)= Σ Ps.a < -e] exp((222/1) thus
PfMa, < - Vtntt
(18)
Similarly, the ME,s are sums of 1-1 martingale increments with values in predictable range of total widths each smaller than v Int, so that, again by the Hooffding- Azuma incquality,
Pf ME, 2 eV8o22n1 < exp( 26
12-222/(111)) 1))
thus PfME. V882 In < 7
Collecting all bounds and recalling that to > we proved the following closed-form finite regret bound, where we recall that to was defined in 13 and where Reg can be bounded by the general In closod-form regret bound
12 proved above:
EP(E(t)
<
3K +
thus
RT
Re, + max Ma (4K + 1).
(19)
aE[K]
(=1
Combine the logarithmic regret bounds 10 and 12 with the upper bound 14 on In to to get closcd-form expression of the final regret.
B Proofs of Theorem 2: Lower bound on the diversity-preservingg regret
The main difference with respect to the classic proof schemes for lower bounds (Lai & Robbins 1985 Graves
& Lai 1997 Garivier et al. (2199) is described in Remark [ eventually, arms A, are played, so that the information gain should be quantified in terms of the A, and is larger than if it was quantified in terms of the distributions P, used; yet, the UFC constraints on the strategies are terms of the distributions 2, used. This leads to the specific constrained minimum on lim inf Rr/ In 7 stated in Lemma
21

---

Published in Transactions on Machine Learning Research (07 /024)
B.1 General lower bound given by constrained infimum
For model D and bandit problem z in with a single optimal distribution (²(2) in P, we introduce the following set of confusing alternative bandit problems:
ALT(U) = E in D (²(2) & Opt(2') and LA vI
V/ = V/2
or (1) = 0 and KL(va, v/2) < }ool
The bandit problems in ALT(2) are such that p² (2) is suboptimal for them, yet, the player cannot discriminate them from y by only playing p" (2), as, for each arm a, either (T = 0 and selecting the optimal probability p²(2) never results in picking arm a, or Va v/ and observing reward associated with does not provide discriminative information.
The proof of the following lemma relies on standard techniques introduced by Graves Lai 1997 This is why postpone its proof Appendix B.3 The linear program defining c(Ext(P), 2)) is over vectors (np).
where p p p² (z), ie., no component (w) is considered.
Lemma 4. For all possibly randoviizd UPC stralegy over D given P,, only picking distribulions in Ext(P), for all bandit problems y in D with a single optimal distribution in P denoted by (²)),
lim inf Rr T-000 In T
2 a ((Fxt(P))
def
inf
Σ
A nz:
and Vv E ALT(E), n2 n₂
Σ n
Σ
(L)),,
PEExt(P) 2 (
Dc Px)()
aE[K] P1))00
2+
We may also prove the following equivalence, which will be the key for both the proof of Theorem and the one of Proposition 1
Lemma 5. Consider a (mean-bounded or mean-unbounded) model D, a Seuerssty-preservivie polylope and bandit problem y in D with a single optimal distribution (²() in P. We have the equivalence:
ALT(Y) 4Ø
c(Ext(PP).) > 0.
Proof. If AuT(y) is empty, then the linear program defining c(Ext(P), is unconstrained, that c(Ext(P),z) 0. If ALT(g) contains a problem g', then since p" pz() not optimal for L,, there exists at least a E [K] such that (() and v/2 + un with KL(Va, v/) +00, and also have
Kmnsx 22) def
KL ) < t00.
aE[K]
P (z)=0
Thus, by the constraint satisfied by y for the first inequality and for the second equality, by substituting < A A for (z), we get
Pa KL(Pa, Xmax(2,K') A
n
Σ
Σ
B ng
(P) (P) ## (v)
[E[R]
&1 KKama (2,22)
PEFxx(P) pøp (v)
Pa (2)==
This imposes the lower bound A/Kmaxxyy v') > on c(Fxi(P))
22

---

Published in Transactions on Machine Learning Research (07 2024)
B.2 Proof of Theorem 2
Reduction argument. We first note that any (possibly randomized) strategy picking distribu- tions in the polytope P can be converted into (randomized) strategy picking distributions only in Ext(P), as required by Lemma 4 This is indeed possible as only the final pure actions drawn matter. More precisely, given a probability distribution Pe over P, we introduce
Step 1:
| pdp(l))=
E Pg(pt) p. and let D(p)) (9g(p:)) DEEx((P)
where the convex weights Pg(pt) exist as P is the convex hull of Ext(P). We interpret the vector p(p)) as a probability distribution over Ext(P). Now, the random variables A, [K] and A [K] drawn as follows, in two-slage randomizations, have the same distributions:
Le Pa then A ~~ P. and
E (p:) then A₁ g
Step 2: Application of the results of Section B.1 Thus, by Lemmas and 5 suffices show that Atr(e) is not emply. Let a [K] such that P. 0. By assumption, P thus Exi(P) put some probability mass this arm of we have Ap > O. Now, by the assumption of unbounded means in D, there exists distribution Ca E D with expectation Na A(p)/Pa- By Assumption
there exists p Ext(P) with Pa >> 0. Since (*(v) is the unique optimal
D there actually exists va D with expectation P/a Ma + A(p)/Pa and such that KL(Va, v/2 < +00.
We denote by v' the bandit problem such that 1/4 va for all k & and whose a-th distribution is 1/2, and claim that z' E ALT(Z). To see this, it only remains to show that (2) is suboptimal for y/: indeed, since (v) = 0 while y and s/ only differ at a for the first equality, by definition of Ap for the second equality, and by construction v/ for the rest,
Ap
w (2) 1²) = (p p) = (p, H)) Pa
(p, v.)
Pa
B.3 Proof of Lemma 4
We believe that offer neater proof than what is usually proposed in the literature.
We fix a possibly randomized UFC strategy over D given P, only picking distributions in Exi(P), and bandit problem y in D with a single optimal distribution (2)). We know that the correct scaling of the suboptimal pulls is at most logarithmic and therefore define the normalized allocations, for all E Ext(P),
E [Ne(T)] In T
Ri In
Ey (V((T) In
UT
that
T
ΣA PEP
(20)
PEP
A UFC algorithm facing the problem e will eventually focus on the unique optimal distribution (z)). Thus, most of its observations will correspond to pure actions E [K] such that p (2) O, which provide no useful information to distinguish y from a given confusing alternative problem y E ALT(E). A measure of the useful information gained is the Kullback-Lcibler divergence
Ir KL (P.,
where Py. and Pr. denote the distributions of rewards Y,...., Yr obtained in the first T rounds (and of the auxiliary randomizations used) when the underlying problems are z and z', respectively.
Step Rewriting Iz. By a chain rule see Equation (8) in Garivier et al. 2019 for the first equality below, by application of the tower rule for the second equality, and by grouping distributions output by
23

---

Published in Transactions on Machine Learning Research (07 (07/2024)
the strategy by their values in Ext(P),
IT Σ By (KL,
Ey
Σ Pt.a (L.) 1/5)
E DEExI(P)
Ey (T)) E
Pa KL((va (21)
aE[K]
Since alternative problems v' E ALT(U) are such that = Va when Po(2) > O, we finally obtain
LT IT
T
Σ aE(K] P. P1(2)=
Pa
(Ka. =
Σ
Pa KL(Va:
(22)
PE)
pEEx P) P#P (2)
KE|K] P. ())
Step 2: UFC entails that Ir is larger than T in the limit. This step is a mere adaplation of the standard proof technique by Lai & Robbins 1985) and Garivier et al. 2019 with distributions in Ext(P) playing the role of arms in the mentioned references. More formally, the crucial observation is that the regret is expressed in (2) in terms the number of plays of suboptimal distributions in Ext(P). Thus, that the strategy is UFC and that p²(z) is, by definition of ALT(Z), suboptimal for v/ entail that, for all 0,
*p [x[A2[T)] o(T"),
well as
Eg Eg (N o(T
(23)
In particular, for all 0, there exists T large enough so that Eg [AV (2) (T)) & TE. We denote by kl(p,g) pln(w/g) + (1 pp) In((1 p)/(1 v)) the Kullback-Leibler divergence between two Bernoulli dis- tributions with parameters p and q. By the data-proceee processing inequality for [0, vJauued Section 2.1 in Garivier et al. 2019) and by the standard incquality kl(p, q) > p pln(1/g) - 2, for all E > 0. for all T sufficiently large,
valued random variables (see
N
(er(w( T
N
(e)(e(())
N (T)
T
I =
2
Ey
Eyr
Ey
In
-In 2,
T
T
Ey
N 2 (g) (T)
S7
where we substituted
23
Letting T + oo then - 0, we conclude to
IT
In - 1.
lim inf
sup inf
(24)
I
In
=>0
Remark 1. The specificily of the selling considered appears in these first two sleps: we use the UFC properly Ext(P) but pure actions As eventually played, so that could lower bound Tr/ In in 22 by
Σ THE
Σ KE[K] PC (2)=0
Pa KL(Pas 1/)
Σ
UT.
KL
Σ [E[K]
Pa Var Σ Dava
PEE
PEEx((P)
aE(K]
where the inequality holds by converily of KL and the fact that U/ = when (v( > 0. The right-hand side above correspords to the measure of information gained when playing distributions p, without observing the pure actions AІ-
22 and 24 as follows. Let e be cluster point of the sequence Ry/ T. cf is the only value, then RT/ In too and the result proved; otherwise, take finite c. We denote by (Tm)ma1
Step 3: Considering cluster points. We combine 20
increasing sequence such that Rym In Tm + In view of the decomposition (20 and since WT...p 2 0 and A(p) > 0 for all p Ext(P) with p # (2),, all these sequences WT... are bounded. Hence, we may extract a subsequence (Tmy)ss from (Tm) such that all sequences TT converge as + 0o, to limits denoted by ng. This only holds for E Ext(P) with p p² p2()), but c ((Exi(P), 2) is only defined based such vectors.
24

---

Published in Transactions on Machine Learning Research (07 (07/2024)
These convergences yield first, by 20 that
RT
lim inf
Ag
7
In
pc Ext(P)
These convergences also entail, together with v ALT(V),
22 and 240,
and the definition of lim inf, that, for any
2
Σ
Pa (L(va.
lim inf
TTT
Σ
Pa (V. 2 1.
(P)
[E[K] P1()==
DEExt(P 2#2 (1)
aE|K] PZ(2)=
2 (2)
Put differently, the limits Up defined above, where p # p* (2), satisfy the constraints the defining infimum of c(Ext(().v). This concludes the proof.
C Proof of Proposition
The proof builds on the proofs for upper bounds (Appendix A] and lower bounds (Appendix B
We provide here some general considerations. The Bernoulli distributions are sub-Gaussian with parameter o² = 1/4. The problem y (Ber(0), Ber(1/2)) Ber(O)) considered has expectations (0, 1/2, 0). The distributions z( = (0, 1 1/2, 1/2) and Pb = (6, 0, - 8) obtain respective expected rewards
(2)
L 2 > 0 L
In particular, given that the polytope Ps considered is the segment between (()) and pa (2) pll) is the unique optimal distribution in P.
the distribution
C.1 Case 5 1/4: In T regret is suffered
The setting of Proposition I satisfies the assumptions of Lemmas and up to considering the reduction
performed in Appendix B.2 It therefore suffices to show that ALT(U) is not empty.
Given that (()) puts a positive probability mass on arms 2 and 3, alternative problems in ALT(e) may differ from only at arm 1. Given that ()( < +oo if and only if μn [0, we have
ALT(y) = { (Ber(u(), Ber(1/2), Ber(O)): 14 E [0, 1) and 2 & Opt(z')
Now, denoting by n' the expectations of v', and given that Py is a segment, we have that the condition 2 & Opt(y') is equivalent to
=
which is equivalent to / > 1/(4.). Therefore,
ALT(y) = {L' =BBrr(K)), Ber(1/2), Ber(O)) 1/(40) < n4 < <1} ALT(Z) is not empty if and only if 8 > 1/4, which concludes the proof.
C.2 Case 8 1/4: bounded regret may be suffered
We the Box-B algorithm with uo 1/2 but make that the upper-confidence estimates of the expectations always smaller than the known bound 1 rewards, i.e., replace Step 1 of Box B by
Va(l 1) (U 1), 1}, V(I - -Vi(()))) Vi) 2, argmax (p. V(I (()=
pεExt(P)
this is the crucial modification for this proof, see 266 for an explanation why is handy. All other steps of the Box-B algorithm remain unchanged.
25

---

Published in Transactions on Machine Learning Research (07 (274))
Step 1: Preparation and adaptation of existing proofs. We borrow several elements from the proof conducted in Appendix A
We consider the same favorable event (() as therein, but only one event in terms of number of pulls namely, we are only interested in arm 3:
E(t) For all a E [K] |mo - -Ra(())) <
2lnt
and g = Na(() > 1/3)
(
max (t).
Lemma shows that P(E()) & 2K for all > 2 e2/8. Given that Pt.3 v min (P3 P8.3] -1/2 by
(1)
definition of Pa as segment and the fact that 6 1/4, we apply the Hoeffding Azuma bound (18 follows: for all > 1,
1 < & P(G()) & PP Ns(1)=
No(1) - Σ
1 - Pa.3 < - & t--1
P
2 6
Since 5 < 1/4, there exists threshold to > such that
6hnt
14 2 lo
8 2
We show below that
vt to, on Et)) (t),),
(g v(t)) > (2.e), v(t)),
thus,
B++1 = =2
(25)
We recall that p()) is the unique optimal distribution for and that the gap of Ps equals 1/4. The property 25 therefore leads to the following bounded regret bound: for all T > to.
(2)
R
1
to 1 Σ P (((()) t=to
& lo 4
(2kr- 3 -11881
to +
K
5.
4
t=fo
Step 2: Proving
25
By definition, we have, (, V(I)) > (2 2)) = 1/4 on E(1), and
2ln (g. k(O) = 8 vi(()) +1- (1-)) Va(t) < < max [N(())11
still on Et),
(26)
We crucially used here the boundedness of (1),,, which we upper bounded by 1. This is fortunate, as arm 1 is only played when the suboptimal distribution p3 picked, so that N, (t) could be small and E(t) would not provide an efficient bound.
Therefore,
(p v(t) > 1 4 and (0f)) k()) & 6 2 6lmt
on Et) ((t,
so that 25 follows by the definition of to.
D Proof of Theorem 3
The proof strategy is similar to that of Theorem with some differences we will highlight: we still work under the favorable event. under which all the confidence bounds are correct, and all arms have been picked a linear number times. Both these events hold with high probability since the probability of picking any arm is lower bounded by some positive constant at all rounds. In that case, the upper confidence vectors (tt) are a good estimators of the true mean payoff vector μ,, and the distributions picked by Setersity-preeerviie UCB get close to the optimal distribution(s). Crucially, we show in Lemma that the differences bet
26

---

Published in Transactions on Machine Learning Research (07 (224))
the expected payoffs of the distributions selected by UCB and the optimal payoff depend quadratically the differences ween (t)) and μ.. This is consequence of the curvature of the diversity-preserve set Pr- Since the error in estimating by (t) is of order Vlnt/t, we obtain per-round regrets of order Int/t, which sum up to In T.
We start with two lemmas, providing first closed-form expression of the optimal distribution (Lemma 6 and second, deducing a quadratic upper bound on the instantaneous regret (Lemma 7
Denote by V = { E Rd f4 =... } the subset of sypected-payoff vectors with identical components. All strategies get a null regret in bandit problems y with payoffi in V. We may therefore exclude this case in the rest of the proof. We denote by 1 the vector B with components all equal to 1, and refer to the model of all distributions over with finite first moment as Dall-
Lemma 6. Consider Pr with r < Fim- For any bandit problem y in Dall with means μ E RR VV V, the optimal distribution in Pr is unique, only depends on Ht, and equals
P.(m)
1 K
+
1p Mavgll (1 Pavg ))
r
where
Pavg
1 K
Σ aE[K]
Me.
Proof. We denote already by P. (1) the vector defined above and prove that it is the only optimal vector; we do via elementary arguments. For any p Pr, the difference between the expected payoffs of p and P, (()) equals
r
H. P. (1) -p) = (1: [1 1p)
+
H4 Mavg1)
K
| Mavg 1|
r
Mave11 1,
-p) +
| 1 1p- Mavg 111 (1 - Mavgl, 11 Mavg1)
K
| Mavg 1
r-
1 p| 0, K
1
(27)
where we used, for the second equality, that both vectors (1/K)1 y and μ - Mavg 1 are orthogonal 1 (as their components sum up to 0), and where, for the final inequality, applied the the Cauchy-Schware inequality:
4 Mavgl 1, K 2) > -11 Mavgl] 1|
| - 2
Now, the inequality (27 is an equality if and only if μl Mavgl and (1/K)1 - p are colinear, that is, if there exists a E IR such that
a
Mavg 1|| (M Mavgl)
By definition of Pr, such a p in P, if and only |al & r. Therefore, final equality to 0 is achieved in 27 if and only if a r, i.e., il p = 2, (H).
The curvature of the Unversity-preservve
set P, guarantees that the best distribution varies smoothly with the expected payoffs. This in turn implies that the per-round regret of the depends quadratically on the difference between the upper confidence bound vector U and ę-at least when U in a neighborhood of μ.
UCB strategy
Lemma 7. Consider Pr with < Film and use the notation of Lemma For any popected-payoff vector EE Rdd \V, define the function
6u Q \ \ - (p,P,(u)
There exist constant By such that
AR YEERKYY V
(n(1) (.( & Bp 1| rl
27

---

Published in Transactions on Machine Learning Research (07 (07/2024)
With this notation, if p, is the distribution output by the unversity-preeevviin UCB strategy at round then the expected per-round regret suffered at time can be expressed as
= E ( ( 2 - )x(w) - E
o (1(( 1))
Proof. By Lemma ou admits the closed-form expression
2 BUERR V
(1, 1) 1/ Vavg 1 K
( v Envg 1)
showing that Uy C at all points y such that Vargl 1 + O, i.c., over RK V. Morcover, by definition of P. (p), the function dp reaches its maximum at μ4, therefore its differential at y is null and the Taylor expansion around of on reads
(p(4) (n(()) + O 1ul1))
Put differently, the function
(x()) - (p)
U
1
is bounded some E neighboourhood μ. Outside of this neighborhood, the denominator ||0 - #ll is larger than E while the numerator is bounded by 2max Therefore, 1i is bounded over the entire R V. 0 aE(K]
We are now equipped to prove Theorem 3 distributions is used now. We adapt several arguments already reviewed in Appendix A and therefore provide a concise proof.
The fact that the model D is composed of o2-
Proof. Consider the favorable events
8 Ind
and g'(t) = Vaa [K], Na(t) >
(1'lmm r)
E(t) Va E [K], |Ma- - -aa(t) &
max (No(1).)
2
For any t, under E(t) (t),, by Lemmal
16o Int t(rim r)
16K B, Int
(,) - p,) = ( - (U((t - 1)) & Bullu llt - 1)11 < B Σ
Fiim
[K)
Therefore, taking expectations above,
16K Byo lnt
+ max Ma (P(E(T) + IP (7((i)
= B( P. (H) p,Y <
(28)
Flim
t
KE|K]
Σ P (E((()) < +00.
We showed already in Similarly, in the argument following 17 introduce the martingales Mast, and note that by we have
11 in the proof of Theorem I that
P Ma.t < t(rim - r)
c')
2
[E[K]
By the Lzuma-Hocffdiie inequality, the sum above is bounded, cf.
18 Sum
28
over to conclude.
28


Reviews:
[{"review_id": "MJjKASXuVT", "reviewer": "Reviewer_SArx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nFairness is an important and hot topic in machine learning area. The studied problem is interesting. The proposed algorithm achieves bounded regret when diversity is desirable.\n\nWeaknesses:\n\n1. The regret lower bound only works for the mean-unbounded cases, which is less common than mean-bouned cases in real applications.\n2. The paper does not provide any empirical results for the proposed methods.\n3. The main paper is not self-contained. I think the closed-form regret bounds should be presented in the main paper. Also, I think the authors may provide some discussion on the closed-form bound. For example, does the regret bound in Thm 1 match that of vanilla UCB when the polytope is $[0,1]^K$? (In this case, the proposed algorithm seems exactly the same as the vanilla UCB)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies diversity-preserving multi-armed bandits. The authors design a UCB algorithm which enjoys bounded regret when diversity is desirable. The paper also present the regret lower bounds for the mean-unbounded cases.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The writing can be improved, see weaknesses 3.\n2. In section 3, the authors may present an example application which is mean-unbounded.\n3. The paper may provide some empirical results of the proposed methods. I would like to see the empirical comparisons among  the proposed method, Celis et al (2019) and Liu et al. (2022).", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4FdR7mGcRR", "reviewer": "Reviewer_wpTj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper provides interesting results that are sound and quite expected, although I have not checked the proofs in detail.\n- The setting considered, although introduced in previous works, seems to have nice applications in the real world.\n\n**Weaknesses**\n- [Related Works] The presented setting has notable connections with regret minimization with expert advice and bandits with mediator feedback. These settings share with the presented one the fact that arms are not directly pulled but a distribution (possibly non-stationary) is placed in the middle. In particular, these works are related since in some cases they are able to show that constant regret is possible, as in the present paper. The authors should include them in the related works section and provide a comparative discussion.\n\n[1] Auer, Peter, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. \"The nonstochastic multiarmed bandit problem.\" SIAM journal on computing 32, no. 1 (2002): 48-77.\n\n[2] Metelli, Alberto Maria, Matteo Papini, Pierluca D'Oro, and Marcello Restelli. \"Policy optimization as online learning with mediator feedback.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 10, pp. 8958-8966. 2021.\n\n[3] Eldowa, Khaled, Nicolò Cesa-Bianchi, Alberto Maria Metelli, and Marcello Restelli. \"Information-theoretic regret bounds for bandits with fixed expert advice.\" In 2023 IEEE Information Theory Workshop (ITW), pp. 30-35. IEEE, 2023.\n\n- [Technical Novelty] The obtained results are, in my opinion, quite expected and this is not necessarily an issue. For the $\\log T$ regret bound, as the authors acknowledge, this can be obtained by running standard UCB on $Ext(\\mathcal{P})$. For the constant regret since the optimal distribution is played for an expected number of times that is linear in $T$, and the optimal distribution plays all the arms with a non-zero probability, we are like in an \"expert setting\" since playing the optimal distribution provides information for all the arms. Is this consideration correct? Anyway, I encourage the authors to revise the manuscript to highlight in the main paper the challenges of their theoretical analysis and the elements of technical novelty over the previous works, especially [2], with which I noted several elements of similarity in the analysis. \n\n- [Computing $Ext(\\mathcal{P})$] Is computing $Ext(\\mathcal{P})$ computationally efficient given a description of $\\mathcal{P}$ based on constraints? Can the authors elaborate?\n\n- [Tightness] As far as I understand, the lower bound of Theorem 2 provides a lower bound that matches the $\\log T$ dependence of the upper bound of Theorem 1. Is there an analogous showing the tightness of the scenario in which the proposed algorithm suffers constant regret? More in general, apart from the dependence on $T$ all lower and upper bounds do not provide the explicit dependence on (i) suboptimality gaps; (iii) number of arms/number of points in $Ext(\\mathcal{P})$; (ii) constants. This does not allow to fully judge their tightness.\n\n- [Presentation] Section 3 seems to include several heterogeneous contributions. It starts with the lower bound but then the role and significance of Corollary 1 and Proposition  1 are not fully clear. Can the authors clarify?\n\n- [Non-polytope $\\mathcal{P}$] The case in which $\\mathcal{P}$ is a general convex set (not a polytope) is just addressed by considering a specific example of  $\\mathcal{P}$. It is not clear how much this example is general or how these considerations/methods can be generalized beyond the example. \n\n**Minor Issues**\n\n- The paper lacks a Conclusion section\n- Assumption 1: I think the second $E(\\zeta)$ should be replaced with $E(\\zeta')$", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the diversity-preserving K-armed bandits, a setting derived from the standard K-armed bandits in which the agent is allowed to play a distribution over the arms chosen in a properly provided set. The authors propose an optimistic regret minimization strategy that, under the assumption that the distribution set is a polytope, plays the extremal points distributions and uses the collected rewards to update the arm statistics. Under the subgaussian distributional assumption, the authors show that the algorithm enjoys logarithmic regret that reduces to constant under the assumption that the optimal distribution plays with non-zero probability each of the arms. A lower bound is provided matching the logarithmic regime. Furthermore, an example of the case in which the set of distributions is not a polytope is provided with a proposal on how to address it, leading to polylog regret.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to Weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ioKzdmolff", "reviewer": "Reviewer_2E9z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n* The problems posed are interesting and well-motivated.\n* The writing is clear and well-structured.\n* The results (if they can be shown to be true) are quite interesting, especially the bounded regret.\n\n# Weaknesses\n\n## Technical Correctness\n\nThe biggest weakness is that there is a mistake very early in the proof of Theorem 1. Namely, the inclusion of events described in (6) is incorrect. While many of the other steps in the analysis are correct, most results depend on (6) at some point, and so it is unclear which results in the paper are true. \n\nWe can see the incorrectness of (6) by a simple example. Say $K=2$ and $\\mathcal{P}$ is just the standard probability simplex. (Any polytope which contains $\\begin{bmatrix}0 \\\\\\\\1 \\end{bmatrix}$ will suffice.) For any $\\Delta >0$, $\\sigma>0$, we can find $T$, $t$, and $n$ such that\n* $1\\le t \\le T$\n* $n\\ge \\frac{65 K\\sigma^2}{\\Delta}\\ln T$\n* $t\\ge n/2$\n* $\\sqrt{8\\sigma^2 \\ln t}>\\frac{\\Delta}{2}$\n\nFor example, if $\\Delta=\\sigma=1$, $T=1000$, $n=899$, and $t=450$ satisfy all the constraints.\n\nSay that $\\underline{p}=\\begin{bmatrix}0 \\\\\\\\ 1\\end{bmatrix}$,  $N_1(t)=0$ and $N_2(t)=t$. Then the event on the left of (6) holds:\n$$\n\\bigcap_{a \\in [K]}\\\\{N_a(t)\\ge np_a/2\\\\}\n$$ \nThe event on the right of (6) does not, since the right is\n$$\n\\mathcal{E}'(t)=\\left\\\\{\\forall a\\in [k], \\sqrt{\\frac{8\\sigma^2 \\ln t}{\\max\\\\{N_a(t),1\\\\}}} < \\frac{\\Delta}{2}\\right\\\\},\n$$\nand the inequality fails for $a=1$.\n\n## Technical Precision\n\nBeyond the error above, there are other instances of technical imprecision. For example, as written, **Assumption 1**  holds trivially, since we could always just take $\\zeta=\\zeta'$. Presumably something else was meant. \n\n## Potentially Outdated Literature Review\n\nWhile the problem is set up nicely, and context is given, the references are generally fairly old. From what I can gather from the text, this manuscript is based on a preprint written no later than 2022 (since it states that Liu et al. (2022) came after the preprint), and the bulk of the references come from prior to this time. My guess would be that many relevant follow-up works have already appeared. While the cited references may be the most relevant, giving an up-to-date discussion would be helpful.\n\nI purposely did not look to see if the discussed works had more recent follow-up papers in order to preserve anonymity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper gives upper and lower bounds for classes of multi-armed bandit problems with constraints on the action distributions. They discuss how such constraints can be used to describe various conditions that could be desirable in applications.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main thing to do is fix the technical errors. \n\nA secondary, but very important change, is to do a more thorough literature review to see what has been done more recently related to these problems.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper revisits the framework for diversity-preserving $K$-armed bandits. It presents a UCB algorithm tailored for this setting, offering theoretical guarantees on its performance. The authors provide regret upper and lower bounds and discuss the application of their approach beyond simple polytopes.

The paper presents valuable theoretical contributions to the field of diversity-preserving $K$-armed bandits. While there are some weaknesses, particularly in "empirical validation and completeness of the main paper", the authors' responses indicate a willingness to address these issues. With the suggested revisions, the paper has the potential to make a significant contribution to the field.
Decision: Accept (poster)