Bandits with Ranking Feedback
Authors: Davide Maran
Francesco Bacchiocchi
Francesco Emanuele Stradi
Matteo Castiglioni
Nicola Gatti
Marcello Restelli
Venue: NeurIPS 2024 poster (2024)
Keywords: Online Learning
Bandits
Primary area: bandits

Abstract: In this paper, we introduce a novel variation of multi-armed bandits called bandits with ranking feedback. Unlike traditional bandits, this variation provides feedback to the learner that allows them to rank the arms based on previous pulls, without quantifying numerically the difference in performance.  This type of feedback is well-suited for scenarios where the arms' values cannot be precisely measured using metrics such as monetary scores, probabilities, or occurrences. Common examples include  human preferences in matchmaking problems. Furthermore, its investigation answers the theoretical question on how numerical rewards are crucial in bandit settings. In particular, we study the problem of designing no-regret algorithms with ranking feedback both in the stochastic and adversarial settings. We show that, with stochastic rewards, differently from what happens with non-ranking feedback, no algorithm can suffer a logarithmic regret in the time horizon $T$ in the instance-dependent case. Furthermore, we provide two algorithms. The first, namely DREE, guarantees a superlogarithmic regret in $T$ in the instance-dependent case thus matching our lower bound, while the second, namely R-LPE, guarantees a regret of $\mathcal{\widetilde O}(\sqrt{T})$ in the instance-independent case. Remarkably, we show that no algorithm can have an optimal regret bound in both instance-dependent and instance-independent cases. Finally, we prove that no algorithm can achieve a sublinear regret when the rewards are adversarial.

Bandits with Ranking Feedback
Davide Maran"
Francesco Bacchiocchi"
Politecnico di Milano davide. maran@polimi..it
Politecnico Milano
francesco. Macchiocchhi@polimie
tranesso.bacchiocchhpppp.
Francesco Emanuele Stradi"
Politecnico di Milano
francescoemaauele..ttoo streeestsaadiii @polimi itt
Matteo Castiglioni
Nicola Gatti Politecnico Milano ni nicola. gatti@polmmi
Politecnico Milano
matteo. pattoo..astiiiiinniipl..........
Marcello Restelli
Politecnico di Milano
rescello.restellll 10l.ippmpliiitt
Abstract
In this paper, we introduce novel variation of multi-armed bandits called bandits with ranking feedback. Unlike traditional bandits, this variation provides feedback the learner that allows them rank the arms based on previous pulls, without quantifying numerically the difference in performance. This type of feedback is well-suited scenarios where the arms' values cannot be precisely measured using metrics such as monetary scores, probabilities, occurrences. Common examples include human preferences in matchmaking problems. Furthermore, how numerical rewards
its investigation answers the theoretical question crucial bandit settings. In particular, we study the problem of designing no-rege nogregret algorithms with ranking feedback both in the stochastic and adversarial settings. We show that, with stochastic rewards, differently from what happens with non- ranking feedback, algorithm suffer logarithmic regret in the time horizon T in the instance-depea eettnce-deeendent case. Furthermore, we provide two algorithms. The first, namely DREE, guarantees superlogarithmic regret in T in the instance-dependent case thus matching our lower bound, while the second, namely R-LPE, guarantees regret of (vTT) in the instance- incennceedddeeddeent case. Remarkably, we show that no algorithm can have an optimal regret bound both instance-dependent and instance-inde independent cases. Finally, we prove that no algorithm can achieve a sublinear regret when the rewards are adversarial.
Introduction
Multi-armed bandits are well-known sequential decision-making problems where a learner is given number of arms whose reward is unknown [Lattimore and Szepesvari 2017). At every round, the learner can pull an arm and observe a realization of the reward associated with that arm, which can be generated stochastically [Auer
2002 or adversarially [Auer al. 1995 The central question
"Equal Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

in mullti-armed bandits concerns how to address the exploratoon/explotatiinn tradeoff to minimize the regret between the reward provided by the learning policy and the optimal clairvoyant algorithm.
In this paper, we introduce a novel variation of multi-armed bandits that, to the best of our knowledge, is unexplored so far. We name the model bandity with ranking feedback. This feedback provides the learner with a partial observation over the rewards given by the arms, More precisely, the learner can rank the arms based on the previous pulls they experienced, but they cannot quantify numerically the difference in performance. Thus, the learner is not allowed to asses how much an arm is better or worse than another. This type of feedback is well-suited for scenarios where the arms' values cannot be precisely measured using metrics such as monetary scores, probabilities, or occurrences, and naturally applies to various settings, e.g., when dealing with human preferences such as in matchmaking settings among humans and when the scores cannot be revealed for privacy or security reasons. This latter case can be found, e.g., online advertising platforms offering automatic bidding services as they have no information on the actual revenue of the advertising campaigns since the advertisers prefer not to reveal these values being sensible data for the companies. Notice that a platform can observe the number of clicks received by an advertising campaign, but it cannot observe the revenue associated with that campaign. Remarkably, our model poses the interesting theoretical question whether the lack of numerical scores precludes the design of sublinear regret algorithms or worsens the regret bounds that are achievable when numerical scores are available.
Original contributions. We study the problem of designing no-regrer algorithms for the bandits with ranking feedback problem, both in stochastic and adversarial settings. In the case of adversarial rewards, we prove that no algorithm can achieve sublinear regret. In contrast, with stochastic rewards, we show that the ranking feedback does not preclude such a possibility. In particular, in the instance-dependent case, we show that no algorithm can achieve logarithmic regret in the time horizon, and we provide an algorithm, namely DREE (Dynamical Ranking guaranteeing regret bound that matches this lower bound. In the mnstance-indepenee crucial question is whether there exists an algorithm providing a better regret bound compared to the one achieved by the well-known Explore-thes then-Commii algorithm, which trivially guarantees an 2333 regret upper bound. We positively answer this question by designing namely R-LPE (Ranking Logarithmic Phased Elimination), which guarantees regret of Ö(VT) in case if the rewards are Gaussian. To achieve this result, we derive several
case,
algorithm,
the instance-independee non-standard results that allow us to discretize Brownian motions. which are of independent interest. These two different approaches leave open the problem of whether there exists an algorithm achieving optimal performance in both the instance-dependent and innteeeeinndeenneent cases. We negatively answer this question by showing that no algorithm can achieve an optimal regret bound in both cases, confirming the need to design two distinct algorithms for the two cases. Finally, we numerically evaluate our DREE and R-LPE algorithms in testbed, and we compare their performance with some baselines from the literature in different settings. We show that our algorithms dramatically outperform the baselines in terms of empirical regret.
Related works. The field most related to bandits with ranking is preference learning, which aims at learning the preferences of one or more agents from some observations [Fürnkranz and Hüllermeier 2010). Let us remark that preference learning has recently gained a lot of attention from the scientific community, as it enables the design of Al artifacts capable of interacting with human-in -not (HTL) environments. Indeed, human feedback may be quite misleading when is asked to report numerical values, while humans are far more effective at reporting ranking preferences. The preference learning literature mainly focuses on two kinds of preference observations: pairwise preferences and ranking. In the first case, the data observed by the learner involves preferences between two objects, i.e., a partial preference is given to the learner. In the latter, a complete ranking of the available data is given as feedback. Our work belongs to the latter branch. Preference learning has been widely investigated by the online learning community, see, e.g., [Bengs et al.) 2018).
Precisely, our work presents several similarities with the dueling bandits settings Yue et al. 2012 Saha and Gaillard [202]] Lekang and Lamperski [2019), where, in each round, the learner pulls two arms and observes ranking over them. Nevertheless, although dueling bandits share similarities to our setting, they present substantial differences. Specifically, in our model, the learner observes a ranking depending on the arms they have pulled so far. In dueling bandits, the learner observes an instantaneous comparison between the arms they have just pulled; thus, the outcome of such a comparison does not depend on the arms previously selected, as is the case of bandits with ranking

---

feedback. As a consequence, while in bandits with ranking feedback the goal of the learner is to exploit the arm with the highest mean, in dueling bandits the goal of the learner is select the arm winning with the highest probability. Furthermore, while we adopt the classical notion of regret used in the bandit literature to assess the theoretical properties of our algorithms, in dueling handits, the algorithms are often evaluated with suitable notion of regret, which differs from the classical one.
Dueling bandits have their reinforcement learning (RL) counterpart in the preference-based reinforce- ment learning (PbRL), see, e.g., NNovoseller et al. 2019) and [Wirth et al. 2017]. Interestingly, PbRL techniques differ from the standard RL approaches in that they allow an algorithm to learn from non-numerical rewards; this is particularly useful when the environment encompasses human- like entities [Chen et al.) 2022]. Furthenmore, preference- of results, ranging from theory [Xu et al.) 2020] to practice [Christiano et al. [017]] Lee et al., [021].
reinforcement learning provides bundle
In PbRL, preferences may concern both states and actions; contrariwise, our framework stateless since the rewards gained depend only on the action taken during the learning dynamic. Moreover, the differences outlined between dueling bandits and bandits with ranking feedback still hold for
preference-basee reinforcement learning, preferences are considered between observations instead of the empirical mean of the accumulated rewards.
Paper structure. The paper is structured as follows. In Section 2 we report the problem formula- tion, the setting, and the necessary notation. Section we study the stochastic setting, that is, when the rewards are sampled from a fixed distribution. In Section 3.1] we present an instance-dependent regret lower bound that characterizes our problem. Thus, in Section 3.2 we propose our algorithm and we show that it achieves tight instance-dependent regret bound. In Section 3.3 show trade-off between the regret upper bound achievable by any algorithm in the instance-dependenn and instance- innaunceeiddeeddentt cases. Finally, in Section 3.4 present an algorithm achieving an op- timal instance- eextance-independeen regret bound when the rewards are Gaussian. In Section 4 we study the adversarial setting, that is, when statistical assumptions case, we present our impossibility result, showing that no algorithm can achieve sublinear regret. In Appendix and Appendix [B] we report the omitted proofs for the stochastic setting. In Appendix C we report the omitted proof for the adversarial setting. Finally, in Appendix [ we report the empirical evaluation of our algorithms.
made about the rewards. In this
2 Problem formulation
In this section, we formally state the model of bandits with ranking feedback and discuss the learner- environment interaction. Subsequently, we define policies and the notion of regret both in the stochastic and in the adversarial setting.
Setting and interaction. Differently from the classical version of the multi-armed bandit problem- see, e.g., [Lattimore and Szepesvari 2017 which the learner observes the reward associated with the pulled arm, in bandits with ranking feedback the learner can only observe a ranking over the interaction to unfold
arms based on the previous pulls. Formally, we assume the as f
(i) At every round t E [I']. where T is the time horizon, the learner chooses an arm ig E A [n]. where the set of available arms and n = |A|] +oo.
iii) We study both stochastic and adversarial settings. In the stochastic setting, the environment draws the reward ry (it) associated with arm i, from a probability distribution vi,- ie.,
ri (it)
Vi.- whereas, in the adversarial setting, re rée))) is chosen adversarialyy by an opponent from a bounded set of reward functions.
(iii) There is a bandit feedback on the reward of the arm é, E pulled at round leading to the estimate of the empirical mean of i, follows:
(i) Ejemi (1)
Z,(t)
"Given n E NN' we denote with in] = (L.... n). Furthermore, given a finite set A. we denote by SA the set containing all the possible permutations of its elements, Similarly, we let P(A) be the set containing all the probability distributions with support A.
3

---

where Wi(i) = ( i and Zt(t) := [W,( The learner observes the ranking
over the empirical means TEEA Formally, we assume that the ranking R2 E SA observed by the learner at round / is such that:
(R,,s) 2 f. (R.,) VI E [T] Vi,j E [n] s.t. i 2 J,
where Re,i E A denotes the i-th element in the ranking R₂ at round e [Z]. Ties are broken in favor of the lower index.
For the sake of clarity, we provide an example to illustrate our setting and the corresponding learner- environment interaction.
Example. We consider an instance with two arms, i.e., A (1.2), where the learner plays the first arm at rounds t = 1 and t = 3, and the second arm at round = 2, such that Ws(1) = {1.3} and (2) (2). Let r1 (1) = 1 and rss = 5 be the rewards obtained from playing the first arm at rounds / and / = 3, respectively, and let (2 = 5 be the reward for playing the second arm at round = 2. The empirical means of the two arms and the resulting rankings at each round E (3) are given by:
(,(11) = 1, f.(2) 0 R. = (1,2) t t= 1 A(1)) 1, (x(2) R. (2,1) 1 t=
(1(1) = 3, f.(2) R, = (2,1) t= 3.
Policies and regret. every round f, the arm played by the learner is prescribed by policy æ. In both the stochastic and adversarial settings, we let the policy be a randomized map from the history of the interaction H (R1, 1,, R2, 12.................. R, h--1) to the set of all probability distributions
with support A. Formally, we let π H + P (A). for E [2], such that i, ~ (lH(-1)). As is customary in stochastic bandits, the learner's goal is to design policy that minimizes the cumulative expected regret, which formally defined as follows:
T
RI (T = E
Σ
rx(
re((t)
where the expectation in the definition of regret is taken over the randomness of both the policy and the environment, and we define i* E arg max 14 with b4 := E [24] for each i E [n]. In contrast, in the adversarial setting, the regret is defined as (r(x) = Σt=rrd re(tt) - re( and we define arg max Et= (i).). Furthermore, from here on, we omit the dependence on the policy selected by the learner π in the regret formulation, referring to (T(x) as Ry whenever it is clear from the context. In the stochastic setting, we introduce the following additional notation. Given an arm i E [m]. we let A μ4 μ4 represent the suboptimality gap of that arm. Furthermore, when 2, we simply refer to the suboptimality gap as A.
As we will further discuss, the impossibility of observing the reward realizations raises several technical challenges when designing no-regret algorithms, as the approaches adopted for standard (non-ranking) bandits may be challenging to apply within our framework. In the following sections, discuss how the lack of this information degrades the performance of the algorithms when the feedback is provided as ranking.
3 Analysis in the stochastic setting
As preliminary observation, we note that optimistic approaches, such as the UCBI algorithm, are challenging to apply within our framework. This is because the learner lacks the information to estimate the expected rewards of the different arms, making it difficult to infer confidence bounds. Therefore, the most popular algorithm one can employ in bandits with ranking feedback is the explore-then-commtt (EC) algorithm (see, e.g., Auer et al. (2002), ), in which the learner explores the arms uniformly during the initial rounds and then commits to the one with the highest empirical mean. However, we will also discuss in the following, such algorithm achieve suboptimal regret guarantees. Thus, in the rest of this section, we present two algorithms specifically designed to and instance-independee
achieve optimal eestance-dependee
regret bounds.
Note that the latter definition is well well-posed as long as (W)( > 0. For each E A and t E [2] such that [W) we let (i) = 0.

---

3.1 Instance-dependent lower bound
In the classical multi- armed bandit problem, it is well known that is possible achieve an instance- dependent regret bound that is logarithmic in the time horizon T. However, in this section, we show that such result does hold when the feedback is provided as a ranking, Our impossibility result leverages a connection between random walks and the cumulative rewards of the arms. Formally, we define an (asymmetric) random walk as follows.
Definition 1. A random walk is stochastic process {Gi)ee such that:
G. G-= G G t21'
1=0
where fee is an ii.dd sequence of integrable random variables, and Blea] is the drift of the random walk.
Before introducing our negative result, introduce lemma that characterizes the average perfor- mance of two random walks with different drifts.
Lemma (Separation lemma). Let {G:HeN, {Gt]te be two independent random walks defined as:
G++1 G ct
and G+++1 G., t,
where Go = Go = 0 and the drifts satisfy E(e|] = E[e]]. for each t E N. Then, we have:
TP vt, EN G./t 2 G/ylt) 2 c(p, 2) 0.
We remark that the exact value of the constant c(p, q) in Lemma depends only on the two drifts and q, well the probability distribution defining these drifts. In the simpler case of Bernoulli distributions, the constant c(p, q) can be derived in closed form, as shown in Lemma 3 in the appendix. The rationale behind Lemma that, when considering two random walks with different drifts, there exists separating line between them with strictly positive probability. Thus, with non-negligible probability, the empirical mean of the process with the higher drift always upper bounds the empirical mean of the process with the lower drift.
We also observe that the cumulative reward collected by a specific during the learning process can be represented as a random walk, whose drift corresponds to the expected reward associated with that arm. In the classical version of the multi-armed bandit problem, the learner can fully observe the evolution of this random walk, while in our setting, the learner can only observe a ranking of the different arms, making it impossible quantify their performance numerically. Therefore, in bandits with ranking feedback, the only way for the learner to assess how close two arms are in terms of expected reward is by observing subsequent switches in their positions within the ranking. However, Lemma| | implies that even if two arms have very close expected rewards, the learner may never observe switch the ranking, as the average mean of the arm with the higher expected reward may upper bound the second throughout the entire learning process. As result, the following negative result holds.
Theorem 1 (Instance-dependent lower bound). Let be any policy for the bandits with ranking feedback problem, then, for any C [0, +oc) > [0, +00), there exists a A >> and a time horizon T > 0 such that RT log)).
3.2 Instance-dependent upper bound
We introduce the Dynamical Ranking Exploration-Exn Extiiiiatttxioiitiininn algorithm (DREE). The pseudo-code is provided in Algorithm I As usual in bandit algorithms, in the first rounds, pull for each arm is performed (Lines 3) At every subsequent round >> n, the exploitation/exploraiii
trade-off' is addressed by playing the best arm according the received feedback unless there is at least one arm whose number of pulls at & smaller than a superlogarithmic function ft) (0, oo) -> RR. More precisely, the algorithm plays an arm at round if has been pulled less than f() times
A function ft) is superlogarithmic when lim inf f())
+oo Coherently, subpolynomial function is
log(t)
$-000
such that for every 0 have lim sup
100 7
0.
I-+30
5

---

(Lines 5 where ties due to multiple arms pulled less than f(t) times are broken arbitrarily. Instead, all arms have been pulled at least ft) times, the arm the highest position of the last ranking feedback is pulled (Lines 6 Each round terminates with the learner receiving the updated ranking over the arms as feedback (Line 9. Let us observe that the exploration strategy of Algorithm [ deterministic, and the only source of randomness concerns the realization of the arms' rewards.
We state the following result, providing the regret upper bound Algorithm I as function of f.
Theorem 2 cunstance-dependee upper bound). Assume that the reward distribution of every arm 1-subgaussian. Let f (0) + IR+ be superlogarithmic nondecreasing function t. Then there is term C(f, A4) for each sub-optimal arm [n] which does not depend on T, such that Algorithm 1 ssatisfies Ry I (1+ f(T)) Ei=1 A; + log(T) Ei=1 C(f.
To minimize the asymptotic dependence on T of the cumulative regret suffered by the algorithm, we can choose as an example f(t) log(t)++5, where the parameter 8 >> O is as small as possible. However, if A; 1, the minimization of ó comes the cost of increasing the term C(ʃ, As) grows exponentially as 8 goes to zero. In particular, the terms C(f. A,) are formally defined in the following corollary.
Corollary 1. Let ō 0 and f(t) log(t))+² +6 be the sperlogarithmic funciio used in Algorithm then we have:
2A, (<((2)43) +1
C(S..))
1-e -A712
We remark that the term C(f, A.) de- A,, suggest-
Algorithm Dynamical Ranking Exploration-Exploiattoo (DREE)
pends exponentially ing that C(f, A,) may be large even not
when adopting values of 8 that arbitrarily close to zero. At this point, natural question arises. Is such an exponential dependence with respect the suboptimality gaps unavoidable, 5:
1: for E [T] do
2:
if S n then
3:
play arm it
4:
else if arm i has played <f(() times then
play i, =
or is it a consequence of the kind of feedback the learner receives? In the next section, we answer this question.
else
play i, = RR-11111 8: end if'
7:
9: receive updated ranking R₁
Furthermore, let observe that Algo- rithm satisfies important properties.
10: end for
More precisely, (i) it matches the instance nestancedeeeedeet regret lower-bound, since f()) be chosen arbitrarily close to log(t), (ii) it works without requiring the knowledge of the time horizon T, thus being an any-time algorithm.
3.3 Instance dependenntindeppendeer trade-off
In this section, we provide a negative result, showing that no algorithm can achieve good performance in terms of both instance- instance-dependent and instance-indeepedent regret bounds, suggesting that the two cases need to be studied separately. Formally, the following thcorem holds.
Theorem (Instance Dependent/Ineppeneee Trade-off). Let n be any policy for the bandits with ranking feedback problem. If satisfies the following properties:
(instance-dependent regret upper bound) Ry S Ei C(A))a
(instance-indepeedden regret upper bound) Ry S nCTB
then, 2x + 3 2 where ax 320 0.
Proof. (Sketch) Let P1 = 0.5, p2 = -05 A, P. 0.5 + A, for some A > 0 specified in the proof. We consider three instances:
p: Vi Be(p1)
Dt Be(p1)
Be(1) V2 Be(0)
p**
U2 Be(pz)
12 Be(pi)

---

Clearly, the first arm is optimal in instances P. PMP while the second arm is optimal in P We then define the event:
E. n (R. (1,2)) T=1
In the first and in the third instances, the event Eg corresponds to the optimal arm being ranked in the first position for the first t time steps, while the opposite holds for the second instance. We observe the following two key points. (1) Since the event E, contains the realization of all the rankings up to time i, no policy can distinguish between the three instances until this event holds. Therefore, we have:
Ep (2) = Ep. (2( Ep- [t)
(2) instance p", the event E, happens almost surely for every I. Therefore, to ensure that the policy achieves an instance-dependen regret bounded by C(1)7 in this instance, we need [T) s CT in all three instances.
The main question at this point reduces to: are CT pulls of the last-ranking arm enough to distinguish between the two instances and p*? With change of measure argument restricted to the first CT pulls of the last-ranking arm, we are able to show that, for a sufficiently small value of A > O, distinguishing between P.P" impossible with strictly positive probability. Then, we can prove that if the previous consideration holds, in the instance P", we have:
Ry 2 2 (TI-
for constant p 0 close to one. This lower bound on the instance-inde mnstance-independent regret entails that 2 2a, or, equivalently 3 + 2a 2
From Theorem we easily infer the following impossibility result.
Corollary 2. There is policy for the bandits with ranking feedback problem achieving both subpolynomial regret in the instance-dependent case, i.e,, for every a > 0, there exists a function C(-) such that Rr Σ C(A,)TO, and sublinear regret in the eetaace-inddpendeee case.
To ease the interpretation of Corollary 2 in the following result, we discuss the mnstance-independen regret bound achieved by Algorithm
Corollary 3. For every choice of 8 > 0 in f(t) = log(t) 1+6 there is no value of n > 0 for which Algorithm 7 aachivves an mastance-indepeneen regret bound of the form Rr S O(T
The above result shows that Algorithm| I suffers from linear instance- instance-independee independent regret in T, except for logarithmic terms. Moreover, the following corollary of Theorem 3 Jnnswers the question raised in the previous section. Indeed, we can prove that the unpleasant dependence on the suboptimality gaps A; is not feature of Algorithm instead, it cannot be avoided until the instance-dependent regret has a good order in T.
Corollary 4. Let w be any policy for the bandits with ranking feedback problem that satisfies and instance-dent deependedd regret upper bound of the form Ry < Si= C(A))((T), where f()) is subpolynomial function. Then, C(A) is super-polynomial in 1/A.
Proof. We prove the opposite implication, namely that if C(A) is polynomial in 1/A, then the mostance-indepenae regret bound cannot be subpolynomial in T. By assumption, in case of two arms with just one gap A, we have:
P RT Σ CAA t=1
which implies that the instance-independers regret can be bounded in the following way
Ry < sup min
Σ CA (OC)))(2),AN))) AT
sup min
A>0
A>0
7

---

Notice that, for A T-1(((+1) the first term is less than CT f(T), while for A S T-1(((+1), the second one less than T Therefore, the full instance-indepeendett regret is bounded by:
Rr S CTT S(T),
which polynomial in T. by contradiction, f(T) were subpolynomial, this bound would be sublinear in T, but this contradicts the result of Corollary 2
3.4 Instance-independent upper bound
The impossibility result stated in
Algorithm Ranking Logarithmic Phased Elimination (R (R-LPE)
Corollary pushes for the need for an algorithm guaranteeing sublinear re- gret in the instance-independeer case.
1: initialize S = [n]
Initially, we observe that the stan- dard Explore-then-Comee
initialize = (G(1/2.1,T)
(EC) al- gorithm [Auer et al.) [2002] can be ap- plied within our framework, achieving
3: for I. E [7] do
4: play it arg minjes (t)
5:
update Z4, (t) number of times i, has been pulled
an (T22/3) instance-independee
6:
observe ranking R2
re- gret bound. In the following, we pro- vide brief overview of how the EC algorithm works. divides the time horizon into two phases as follows: (i) exploration phase: the arms are pulled uniformly for the first m π. rounds,
7:
if min.es (t) E C then
a log(minses 2,(0))
(T)
9:
S 5 F(
10:
end if
11: end for
where m a parameter of the algorithm one can tune to minimize the regret: (ii) commilment phase: the arm maximizing the estimated reward is pulled. In the case of bandits with ranking feedback, the EC algorithm explores the arms in the first m 7 rounds and subsequently pulls the arm the first po- sition of the ranking feedback received at round t bandits, the best regret bound can be achieved by setting m [T2/3], thus obtaining an (T2333 regret upper bound. We show that we can get a regret bound better than that of the EC algorithm. In particular, we provide the Ranking Logarithmic Phased Elimination (R-PLE) algorithm, which breaks the barrier of ((T2/3) guaranteeing ((VT) regret bound when neglecting logarithmic terms. Due to the mathematical instruments involved, the proof of this regret bound only holds for the case of Gaussian rewards, as the ones presented in similar setting by Garivier al. (0016) The pseudocode of R-PLE is reported in Algorithm
n. As is customary in standard (non-ranking)
In order to proper analyze the algorithm, need to introduce the two following definitions. Initially, we introduce the definition of the loggrid set as follows.
Definition (Loggrid). Given a, E R.s.t a < and constant value T > we define:
LG(a,,,T) [1>))t(1>>) A [log(T)]' Vj =, O,..., [og((T)) [oo__,,,LLoo[(]]]]
Next, we give the notion of active set, which the algorithm employs to cancel out sub-optimal arms. Definition (Filtering condition). Let be the active set of the algorithm, at a certain timestep. We say that a timestep E [T] is fair if all active arms have been pulled the same number of times times. In any fair timestep t E (T), we define the active set Fe(C) as the set of arms such that:
Fi(C) = VESS Σ
> (+())
sL. T fair
This condition will be called filtering condition.
Initially, we observe that R-LPE differs from Algorithm| 1 as it takes into account the entire history of the process, not just the most recent ranking R,, It also requires knowledge of T. We denote with S the set of active arms used by the algorithm. Initially, the set S comprises all the possible
8

---

arms available in the problem (T) Furthermore, the set which drives the update of the decision space S. namely C. is initialized as the loggrid built on parameters 1/2, 1, 7 (Line At every round / E [T]. R-L.P, chooses the arm from the active set S with the minimum number of pulls, namely E [n]] s.t. (z( is minimized (Line 4 ties are broken by index order. Next, the number of times arm & has been pulled, namely Z,, (f), is updated accordingly (Line 5). The peculiarity of the algorithm is that the set changes every time the condition min; (t) E is satisfied (Line 77.
When the aforementioned condition is met, the set of active arms S is filtered to avoid the exploration on sub-optimal arms, Precisely, S is filtered given the time dependent parameter a (Lines 8 D. We state the following theorem providing an instance- inneeeeddddeeeeetntt regret bound to Algorithm
Theorem 4. In the stochastic bandits with ranking feedback setting, when the noise is Gaussian, Algorithm 2 achieves RT S 62m (T)
Proof. (Sketch) To prove the theorem, we define, for every pair of indices i.j E [n], the event:
Ex =
Eig 0
Pj
p > eb
else,
where Ei corresponds to the event in which arm i eliminates arm j at some point in the process, while vb is a constant defined in the following. The probability that at least one of these events
holds is bounded (by Lemma 6 and employing a union bound) as P(V) = (U (Urgssee(m) DT) S
O (n² log(())T-11220--) since their number is at most nnn - 1)/2.
Therefore, if the complement of the event vi holds, there exists arm it with a gap (w...t. the first arm) less than (n - 1) L), which is not eliminated until the last round. This is because at most n - 1 climinations can happen, and if the complement of the event v/ holds, such climinations concern pairs of arms with a difference in mean of at most v6.
Since the arm i* E [n] is not eliminated until the last round under the event TC, the probability of event E corresponding to the event in which the suboptimal arm E [ri] survives for more than (log(T))T4/2 A.A pulls, is bounded by 27/1/2 thanks to Lemma 7 As a result, employing a union bound over all possible values of i", we can say that the probability that any event EA with i* E [n]] occuring is at most 2(n 1)22-//2, Thus, fixing vp (2 - 1)) ensures that A > A,/n, which entails (TZ 2 ((oog(T)T1/²A77) S On log(7))7-11>> and thus, it holds E Z,
O log(T))T--/3) Finally, using the Regret Decomposition Lemma Lattimore and Szepesvari 2017] we can conclude the proof. Proving the two lemmas, however, is not trivial since it requires to see the whole process discretization of a biased Brownian motion,
and then applying results for this kind of stochastic processes.
At first glance, the result presented in Theorem 4 may seem unsurprising. Indeed, there are several elimination algorithms achieving ((vT) regret bounds in different bandit settings (see, for example, [Auer and Ortner 2010, Lattimore et al., 2020 Li and Scarlett 2022]). Nevertheless, our setting poses several additional challenges compared to existing ones. For instance, in our framework, it is not possible to rely on concentration bounds, as the current feedback is heavily correlated with the past ones. This is precisely the reason for the anomalous growth of the regret in terms of the number of arms n: due the extremely correlated feedback, two union bounds are necessary trough the last proof, and this leads to an increase in the dependence of the order of n. In the impossibility of using concentration inequalities, our analysis employs novel arguments, drawing from recent results in the theory of Brownian Motions, which allow to properly model the particular ranking feedback.
4 Analysis in the adversarial setting
We focus on bandits with ranking feedback in adversarial settings. In particular, we show that no algorithm provides sublinear regret without statistical assumptions on the rewards.
Theorem 5. In adversarial bundits with ranking feedback, there exists a constunt E (0, 1) such that no algorithm achieves o(T) regret with respect to the best arm in hindsight with probability greater than 7.
9

---

Proof. (Sketch) The proof introduces three instances in an adversarial setting in way that no algorithm can achieve sublinear regret all three. The main reason behind such negative result is that ranking feedback obfuscates the value of the rewards as not to allow the algorithm to distinguish two or more instances where the rewards are non-stationary. The three instances employed in the proof are divided into three phases such that the instances are similar in terms of rewards for the first two phases, while they are extremely different in the third phase. In summary. the learner receives the same ranking when playing in two instances with different best arms in hindsight, it is not possible to achieve a small regret in both of them.
Acknowledgments
This paper is supported by the FAIR (Future Artificial Intelligence Research) project, funded by the NextGenerationEU program within the PNRR-PE-AI scheme (M4C2, Investment 1.3, Line on Artificial Intelligence), by the EU Horizon project ELIAS (European Lighthouse of AI for Sustainability, No. 1011 101120237) and by project SERICS (PE00000014) under the NRRP MUR program funded by the -GEU..
References
P. Auer, N. multi-armed bandit problem. Proceedings of IEEE 36th Annual Foundations of Computer Science, pages 322-331, 1995. doi: 10.1109/SFCS.1995 1995.49488..
Y. Freund, and R.E. Schapire. Gambling in rigged casino: The adversarial
Peter Auer and Ronald Ortner. Ucb revisited: Improved regret bounds for the stochastic multi-armed bandit problem. Periodica Mathematica Hungarica, 61(1-2):55-65, 2010.
Peter Auer, Nicolo Disachia and Paul Fischer: Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-255 256, 2002.
Paolo Baldi. Stochastic calculus. In Stochastic Calculus, pages 215-254. Springer, 2017.
Viktor Bengs, Robert Busa-Fekete, Adil El Mesaoudi- Paul, and Eyke Hüllermeier. Preference-based online learning with dueling bandits: A survey. 2018. doi: 10.48550/ARXIV.1807.111 11398. URL
httpps://aaxivv..oo
abs/ 1807 11398
Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-lonn reinforcement learning with general function approximation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 3773-3793, PMLR, 17-23 Jul 2022. URL htttps///proceeding.. mlr. wwwppreessive.... httpe//112ccchheeeeee html
Provably efficient preference-
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2017. URL httphtttt////aiiv org. www./oggg/////oooooeeeeee////ooo abs/1706 03741
Johannes Fürnkranz and Fyke Hüllermeier, editors. Preference Learning. Springer, 2010. ISBN
978-3-642-1412249, doi: 10.1007/978-3--44-444333 URL 978-3-642-18 14125-6
Aurélien Garivier, Tor Lattimore, Emilie Kaufmann. On explore-then-commit staarrtennecommiiittttaiiiiieee Ad- vances in Neural Information Processing Systems, 29, 2016.
Tor Lattimore and Csaba Szepesvari. Bandit algorithms. 2017. URL httphh:://too-laatiiim com/ http///wwlolddd///mm boardds/booo/bbaa pdf
Tor Lattimore, Csaba Szepesvari, and Gellert Weisz. Learning with good feature representations in bandits and in rl with generative model. In International Conference on Machine Learning, pages 5662-5670. PMLR, 2020.
Kimin Lee, Laura Smith, Anea Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning, 2021. URL https: //tpx//rxvvvv/// org. absp/11../222...././ 03026
10

---

Tyler Lekang and Andrew Lamperski. Simple algorithms for dueling bandits, 2019. URL https //arxiv. org/ abaaaabb....0../0 07611
Zihan Li and Jonathan Scarlett, Gaussian process bandit optimization with few batches. In Interna-
lional Conference on Artificial Intelligence and Statistics, pages 92-107. PMLR, 2022.
Ellen R. Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel W. Burdick. Dueling posterior sampling for preference-bar reinforcement learning, 2019. URL btttts::://arxt org. abs/ 1908. 01289
Aadirupa Saha and Pierre Gaillard. Versatile dueling bandits: Best-of-both-world analyses for online learning from preferences, 2022. URL httpp:///axxx wwwiii.oog//bbba wrggab////02// /ttp.///2....000
Lajos Takács. On a generalization of the arc-sine law. The Annals of Applied Probability, 6(3): 1035-1040, 1996.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. survey of preference- based reinforcement learning methods. J. Mach. Learn. Res., (1l):4995-49900 jan 2017. ISSN 1532-4435.
Yichong Xu, Ruosong Wang, Lin F. Yang, Aarti Singh, and Artur Dubrawski. Preference-basee reinforcement learning with finite-time guarantees, 2020. URL http:ttts:///axii.orrrr /bggg//bbb/0066 08910
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits
problem. Journal of Computer and System Sciences, 78(5):1538-153 1556, 2012. ISSN 0022- 0000, doi: htt.a//co.rrttcoe///////22000110 com/science 2011.
URL /httts::///stt sciencedirect JCSS Special Issue: Cloud Computing

---

A Proofs of instance dependent stochastic analysis
A.1 Proof of instance dependent lower bound and lemmas
Lemma (Separation lemma). Let (G:hIe, HGt be two independent random walks defined as:
ét G = + et
and G'+1 = G/ +
where Go = C6 and the drifts satisfy Elec] = > q = B[e[] for each E N. Then, we have:
vt, t' E G./t 2 G2,lt) 2 clp, 4) > 0.
Proof. Let us consider the random walk defined as follows:
G G, + p+
2
Since E [cc - 2
- > 0, and observing that random walk with a non null drift is transient, we know
that there exists a constant ei (p. q) > such that:
G 0, vt > 0) = cp.. > O.
On the other hand, we observe that the random walk:
1+ = G + 4 p 4
2
satisfies the opposite inequality since Ele' p+g 0. Therefore, for the same reason above, we have:
Gi < 0, 14 0) c2 g) > 0,
for constant C2(p. > O.
Thus, since the two processes are independent, we have:
P
n ( 0, G,44 < <0t) 0. f,1==
which entails:
x (
(G. > 0, C <0)
411=1
= =P( 0
ptq 2
> 0, G. 1 - <
(G,
411=1
=P(
ptq
G.lll < p ptg)
(G//l>
2
W'==
<P n (Gi/t> tW1=
(o/tt>,,11)
The latter inequality can be casily reformulated as in the statement of the lemma, concluding the proof.
In order to prove the lower bound, we will also show the following lemma.
Lemma 2. Let (Xi) (Xi)een be a sequence of i..d. Bernoulli random variables. Then, for every event E E Fur where F., is the filtration generated by X1,... Xu, we have:
P 1-p)"
IP Px Be(p) S IP Px.....x X. Be(r)) (E) max
( 1 1-
12

---

Proof. Without loss of generality, let us assume that p' < p. Then, we have:
Belp)
{
1x(x)
II pT
p) p) 2id
10.13"
1x(2)
II 1=1
( (1 - -p')1-add1
(2)
(0.13"
[0.11y
1 2
II 10
2')2-zo
Px X P.....N Baee
where Equation 2 follows from the assumption that p' <. The other way round can be proved substituting p and p'. This concludes the proof.
Theorem (Instance-dependent lower bound). Let be any policy for the bandits with ranking feedback problem, then, for any C [0, +00) > [0, +00), there exists a A > O and time horizon > such that RT Cog))og(T))
Proof. Let p1- 0.5, p2 0.5 A, p2 = 0.5 + A, for some 0 < A S 1/4 (independent on T' that satisfies:
20(1) log(1 4A) 2 -1/2.
We consider the following three instances:
V1 Be(pi)
p 11 Be(p1)
VI Be(1)
P:
p
D2 Be(p2)
F2 Be(p))
122 Be(0)
Clearly, the optimal arm is the first in instances P and p",, while the optimal arm is the second in instance Pt. By contradiction, we assume that, for every time horizon T 0, exists policy NI (+)H)) for each t < T satisfying, in all three cases, the following condition:
Rr(xr) S log)T)
We also define the following event about the rankings received by the learner:
E, = n (R, (1,2)).
The event E, can be interpreted as "up to time f, the learner has always observed the ranking (1, 2). Note that by applying Equation to the particular case of instance p** we have:
log()) 2 Ry (TT(("" = Ep-1[22(0 = Ep- (Z2(')) ET),
where the last step holds since in instance p** the event Ey holds almost surely. Therefore, have:
E (2--) < log(T).
Let N. Then,we we have:
Pp(Z2(T) < E = Pp.. (Z2(T) < Er)rr)
> C(1) log(T)
h
We note that the above equality holds since, under the event Er, no policy distinguish between the two instances n.* and P. On the other hand, the above inequality follows from Markov's 's theorem.
Furthermore, we notice that the event (7) < h3 is contained in the a algebra generated by the first h pulls of arm (and all the pulls of arm 1, but this irrelevant since vi corresponds to the same distribution in the first two instances). Therefore, for all h > O, from Lemma 2 we have:
0.5 A
Pp- (T) h)
Pp (Z) < h)
0.5 + A
13

---

0.5 A
Pp (Z) < Er)
0.. + A
0.5 A A (P)(T( < </ (pr(
(PPr))
0.5 + A
0.5
h
((1) (1-C())(Og(T) x(A)
0.5 +
h
Where x(A) == infe>o (P,) > 0 thanks to Lemma
Here, for every 1/4 2 I > 0, the inequality 0.5 2 (1 4x) 2 0 holds. Thus, for all have:
0.5
h
2 (1-44) (1-44))
0.5 +
Therefore, taking (1( lTg(T), we have:
0.5 A
h (1 (1- (og() k((A)
(Zz(T) < <h)> 2
0.5 + A
h
(1 x(A)
20(1)log(21
(1-42)
x( (og(())og((11))))
Thus, can lower bound the regret in case of instance P follows:
(P((T((P") 2 ARp- [(T Zz(T))]
2 ATT h-Ppp (Zz(T) <)
A(T - Pp < (0(1))
(())(( (0( (200))))
2 (())(( 2C(()) (0)))))))))
2
which grows polynomially with time. At this point, Equation would give, for every T > 0, the following:
log)) 2 <(A)A(T - (0g())ogg(T) (T) 1/2 1/12
Ω (M(A)A 12/2)
2
Clearly, for T sufficiently big, the above statement false, concluding the proof.
A.2 Proof of instance dependent upper bound
Theorem 2 (Instance-dependent upper bound). Assume that the reward distribution of every arm 1-subgaussian. Let f (0,o0) oo) R2 be superlogarithmic nondecreasing function Then
there is a term C(f. A:) for each sub-optimal arm i E [n]] which does not depend on T, such that Algorithm I satisfies Rr < (1 f(T)) Σi A, log(T) Σ C(f. At).
Proof: Let it E [n] be the optimal arm. For any suboptimal arm E [n]. we have:
([2 = Σ P(i, i)
==1
T
11 Σ
P(i, 1 f(7)
Σ -=
= -, f(7)), (4)
where we let i, E [x] be the arm pulled at time E [T]. We split the proof in two parts, providing bound for each term defining Equation
14

---

Claim 1: The first term of Equation decreasing, we have:
is bounded by f(T). Indeed, notice that, being f(-) non-
f())
Σ 1
-
i
p=
T S(T)
Σ
1(, = i, 11)=) -((==)(
7=1 p=l
((2) T
Σ Σ 1=1
= p) - S(T). 2( i,
Claim 2: The second term in Equation (4) bounded by C(A,) log( for some C(A:). We know, by design of the algorithm, that the arm E [n] can be pulled only if:
1. It has the highest empirical mean.
2. Every other arm has been pulled least f(T) times, including arm i e [n].
We define the event:
E,s := (Z+(t) 2 f(t)}.
Then, have:
= -)) P(F) ff(),, Bi,, Er
which can be true only at least one of the following holds:
1. (i) > 14 + 4,2,, which, when intersected with Eir,, holds thanks Hoeffding's inequality with probability at most:
(1))(1) > as + E,,) v
Σ
P(f+ > 44 + A./2, =
(2f(r)
e S(T)
1 1 e A?/2
(2f())
2. Fr fi") <M. -4/2, which, when intersected Ei also true with the same probability as above, thanks to Hoeffding's inequality.
Therefore, have proved that:
T Σ T=1
T P(i, = i, 2,(T - f())) < 2(1 -99/---
s(r)a?
< 2(1 - -2//2-
T=l
Log(T))
log(T) (T)
We show that there exists constant Co(A,) < too such that (+) Co(As) in the equation above. We start observing that, since f(.) is superlogathmic, we have:
f(WA? Fog((
lim sup te (t) s0042
lim simsupte te
lim sup t
6->00
15

---

5(6)
Thus, we can find to such that
2
1 for all 2 to, so that:
S(+)4
D 2r=1
fit44
6
Mija
sip t-000
-r=1" log(1)
limsup 10000
27=1
lim sup 10000
Tato
21.
E'=1 1
D
1
Σ 7=to
1
Thus, since a sequence with finite limit superior is always bounded, we have:
Co(A.) == 2(1 - ------1---- sup
r=
+00,
2>1
log(t)
proving that for every suboptimal arm i E [x]] the following holds:
B[Z((2) < 1+ f(T) + Co(4))
Finally, defining C= A,Co(A)) we have:
12 Rr Σ (T))
concluding the proof.
Corollary 1. Let 5 > and s(() = log(l)2+6 be the sperlogarihmmic function used in Algorithm
then we have:
20, (c(19143)21)) +1)
C(f,)))=
1-e 47/2
Proof.
Let to > 1 be smallest integer such that:
lngie) 1
for all / 2 to
By rearranging the latter inequality we have that:
to = (e((3\/)))) 7
From the proof of Theorem
have C(Ai,f) (Ai), where:
Kr)a
Σ
Co(A.) = 2 221-==
sup
t>1
log(t)
Furthermore, we let L. be the integer satisfying:
Σ1=
f())
log(()
log((+)
for all the integers / 1. Notice that such value always exists, the supremum limit of the above quantity is finite. Therefore, we have:
(6(A.))og(tt = 2
sir)
7=1
Σi= f(T)2
=2(1 - -
log(t.)
log(t.)
1o
s(r)a
s(r]a? I
< 2e11---
D1=
log(t.)
log(t.)
16

---

Where the last step is due to the fact that for to > we have Σi= - log(t.).
Furthermore, thanks the definition of lo, have:
Σ siris
S1.
-r=to 1
With this consideration, we are able to conclude:
r e S1)
Σ
Co(A.) s 2(1 e -1/2)-1
2
ltg((+)
E
1
$ 2 - -1/
to
+1 1
log(t.)
2 (011----1/2)--- + 2(1 1 (22/27)
Where in the last step have used the fact that log(t) for 1. > 2. Recollecting all the terms have:
C(A,, A 22, 1 -e +);
concluding the proof.
B
Proofs in the instance independent stochastic analysis
B.1 Instance dependenn/independeal trade-off
Lemma 3. Let us define a random walk
{1 {
with prob. p with prob. p
GG+1 G. Et
with Go 1 and p= 1/2 + A/2 0.5 0.5 for some A 10. Then, have:
UIG ( )1144
Proof. We define:
Ja P(Go = 31 G. =
which satisfies, for 0, the following recursive equation:
= + (1 - p)
with f. = 1 for < 0. The equation corresponding to the aforementioned dynamical system is:
(1 (1-p)A² X = 0.
The two solutions of the above equation are:
11,2 + /1 4p(1 p)
2(1 p)
Thus, for all n > O, we obtain:
1 +
I 4p p)
1- 4p - p) 2(1 p)
fn A
+ B
2(1 p)
where (otherwise, the equation does not define probability) and B (since f1 for + 0). Therefore, from the definition of we get:
1
4p(1 p)
fn =
2(1 - p)
17

---

4(1/4 2/2/-2/211/2+2///
1+A
(1-A)
1+
The lemma holds by setting 1.
Theorem 3 (Instance Dependen//ndependen Trade-off), Let n be any policy for the bandits with ranking feedback problem. If satisfies the following properties:
(instance-dependent regret upper bound) Ri S Ei=ICh C(A,)T
(instance-independent regret upper bound) RT S nCT8
then, 20 + 3 2 1, where 0 O.
Proof. Let pi 0.5, P2 0.5 A. p2
0.5 A. Let consider three problems: p
:: v1 Be(p1)
PI Be(pi)
#1 Be(1)
p
U2 = Be(p2)
M2 Be(pt)
U/ Be(0)
Clearly, the optimal arm is the first in instances P and p while the optimal arm is the second in instance p
Let us now define the event:
F2 n (R. (1,2))
By assumption, the policy has sub-7 instance-dependent regret. Therefore, in instance p"*, we have for all 1j the following:
Ep.- (Z2(T)) ET!
Ep., [Z2(T)
Ry
lim sup
lim sup
limsup T
0, Ta+o
T T
Ta+n
T-400
Ta+0
since in the instance pr* the event Er holds almost surely.
Under this event, no policy can distinguish between the instances. Therefore,
Ep [p((z(t)|Ed)
Ep., [t2
lim sup
lim sup T-100
6+00
Ta+n
Ta+n
30 10000000 E7(Z2(1) S Ct
Where [p[: is the expectation over the random variables of the rewards in instance P. Therefore, by Markov's 's inequality, we have:
(P(Z2(T) 2CT Ep(Z2(T)|ET)
CTa+n 1
2CTa+n
2CTa+7
Now, note that for every h: > 0, the event (Z2(T) < h} is contained in the a alageerra generated by the first i pulls of arm 2 (and all the pulls of arm but this is irrelevant since arm corresponds to the same distribution in the first two instances). Therefore, thanks to Lemma we have:
0.5 A h
Wh 0
Pp- (Z2(T) <1)>
PP((Z(T) h)
0.5
0.5 A
h
(P7(Z2(T) S h, ET)
0.5 + A
0.5
Pp (Po(Z2(T) S (MET))
0.5
By the previous step, have (P)
2 +1
ET) 2 1/2, so that:
1 1/0.5 0.5 - A 20
Pp.(Zz(T) < 2CT
>
Pp(Er),
2 0.5 A
18

---

while, thanks to Lemma
3 we have:
1- A
(p(ET) 21
2 2A,
1+ A
meaning that:
0.5 - A
2CT
2A
Pp. (Z2(T) S 2CT
0.5 0.5
1+A
We use this result to provide lower bound for the regret in the instance- iettanc-inddeeedeett case. Analyzing the instance independent-regget, by definition, we have fix T as time horizon and let the arm gap A my depend on T.
Let us now fix 1. With the choice A = Pa & 1/4 for sufficiently big T), we have:
1/0. 0.5 - pa
2CT
Pp. (Zz(T) < ))
2
2T pa
2 0.5 + T pcx
20T
~ po
2 47 pr
1 T pa
2CTa+ n T --pox
47 pa)
(5)
2
At this point if we choose 1j = al(p 1)/2, the following fact holds:
lim T-300 (I (1-47-00)
2CT
(1 4y
Cy
pa
lim -
-11/2-o//2)
Cy
lim y-0
1 4y
(1/21e/3)
Cy
lim y-
1
4y
where in the first equality we substituted y = To Here, second exponent is strictly positive.
-(1/2be/2)
p-1 20
where the
-1/ Cy "0
lim (1 11-47-0 pp- 2CTat T0000
(11 (1-44)
lim y00
lim +
(1/e
8-1
=1
This limit shows that there is Cp > O such that:
(1 -1-47 4T
2CT
2 Cp
when 7' > 0 is sufficiently large.
Substituting this property in Equation we get:
Pp- (Zz(T) S 2CT0(1/240/2)) > CpT-pa
2
holding for every p > 0 and sufficiently large T
Thus, with A = po we have:
Ep. Rr 2 A(T - 2CT
< 2CT001/200/2)
Cp Tpp-2pp 2pa
Cp ppp-2pp
2 2 2
T
4
for all T Osufficiently big.
Therefore, for 3 <<1 1 2pa it is not possible to have an instance-independent upper regret bound. Since this is valid for every p > 1, we can also extend the result to any < 2a, which leads to the conclusion that the necessary condition to satisfy both:
19

---

(instance-dependent regret bound)
RT
C(A.)T
AA 0
(instance-indeeeedeet regret bound)
RT S nCTS
for the same policy π is:
2a + +
B.2 Proof's of instance independent regret upper bound
To derive the final instance-independent regret bound, introduce some results from the theory of stochastic processes. The following subsections are therefore devoted to developing all the necessary results to prove the final regret bound of the algorithm.
B.2.1 Discretizing the Brownian motion
In this section, we prove some results about the relationship between random walks and Brownian motions, that will be crucial in the proof of the regret bound. For this scope, we will introduce this quantity:
B + tyt < 1/ - S. (()o..)
corresponding to the Lebesgue measure of the set (t E [o, 1]: B, + tuo < n) We start with a lemma that bounds the increments in standard Brownian motion. Lemma 4, Let (Beheejo11 be standard Brownian motion. We define:
4 = [i/n, (i + 1)/n),
for all i E to...., 1}. Then, for every 17 0, we have:
P sup
(sup B₁ - B1/x)= 2 = Ie1,
inf
(inf B, - Beja)) S -1) BEl.
iE(0......-1
2 n2 exp
n'
2
0/O 2n
Proof. We notice that the Brownian motion satisfies:
IP
sup
(sup B. - Bi/r) 11 = P
sup B₁ - Bi/n =n
iE iE(....,Mn-1)) tel,
i= tE,,
P
sup B, - Bi/r > 11 tel,
2IP (P(Bet4)/n - -Bijn 11)
20,(N(0, 0² 02/n) 1)
2exp
n' 2
n/eexp
n2n (-1'2
2
1/ / 2nt
n/o22T
20

---

where the second equality holds from the reflection principle (see [aldi 2017), ) and last inequality holds since is well known that:
exp(-u2/2,8g
(O B2) > y)
/ 2
for tail bound Gaussian distributions. In the exact same way, we can prove that:
c/Tcxx 2n²
P inf (iE(0...,,11))
(inf B, BBy/a) 5 -1) TET,
17 1/ov22
Together, the two results imply the thesis.
We ready prove theorem that links Brownian motion and random walks in terms of the probability that each of them stays in the interval [0, co).
Lemma 5 (Discretization lemma). Let (G...et,....n- be Gaussian O-mean unit variance random walk, J E IR, and (E0,) standard Brownian motion. Then, for every (0, we have:
P P( + tuo n/ 5) P(n,n) S
Σ 10) (G; + ip.) sn
< P ( ([B + tun -nj > + P(n. m))
and,
1
P F( + tuo S Snl s P((nn) S
Σ 1 (-c,00 (G: + ip)) S sn
1=
< P B. + 1p0 S -nl S 8) + P(n, 1)),
3vTexpp-xntn and p4 nmp. 7/
with P(n,n) n)
Proof. We only prove the first part, as the second one follows trivially by substituting:
+ 1 - 8, + M4, G + - G,, B + B,.
Let (Bt)ee(o, be standard Brownian motion. We define:
I = [i/n, (i + 1)/n],
for all {O..... 1}. Furthermore, we set μ40 Vny. With this definition, we have the following set inclusions for any & E [0, 1] and 1 > 0:
B, + two > nl > = { 1(n,0) + THO) dr > >3}
1(noo) + THO) dr >
sup 1 ((no)) + THO) > sn
Bi/n + i =1 > U
100,00)
sup
sup B, Basn 2 2n) ArEl
iE iE
Moreover, using the same steps above, it also holds:
BB. tpo > /n > s)
1 ( (Br + +To)) dr >
2
n
(00,00
Bi/n + i [10 > sn
inf
(inf Br - BBJr) 2 Bejn)2 -01 EEL.
ie (E(0....
21

---

Now, note that the random variable Bi/n for each E [n] has the same distribution of G Vn, thus:
P
Σ 1=0
(0,00)
Biin + i =u0) > sm
= P
10,000
VVnBe/n + via > sm
P
10,00)
(G; + ip) >
1=0
Therefore, by union bound, we have:
( tuo
7l s)
1 (0))0) (G)+ip) > sn
+P
sup
sup Br Bijaa 21
(E(O
VEl
and,
P ([B tuo -n/ 8) > P
Σ (0,) (G, ip)
-P
( inf iE IEL.
(inf B, Bejn)) -m)
The proof is completed applying Lemmal 4 and reordering the terms.
Corollary 5. Let (E (0 be Gaussian O-mean unit variance random walk, and E IR. Then, for every E (0, 1), we have:
P ( Br + 410 v
2log(n) vn
V2
[-00,0] (Ga + ip) s sm
E
<
nn log
m
P (
Br + +400 2log(n))
v2
8
+
vn
nn (og(n)
where μo VTL
Proof. is sufficient to make the substitution:
log(n)
7 vn
in Lemma 5 Then, we have:
P(n.n) nex (-722//2)
2 2m
expexp ( log(n)2)
log(n)
/2T
Vn
2n
logoon (n)))
22 / knllgg())
log(n) 22T
concluding the proof.
B.2.2 Proofs of filtering inequalities
the proof of this subsections will be based on the following very powerful result, which studies the time spent by a Brownian Motion with drift in the half-line [0, oc).
Theorem (Taáács 1996). Let B, be a standard Brownian motion on E [0, 1]. and let us note as the Lebesgue measure of set. For no R and r > 0, we have
P P[B + tyo S En/S < (1ss))2) =2
+ #o
VI-7
22

---

p(nn/ VT pOVT)
un 4 (-n/ -oon//-HooF)/
HoV7)
dr,
VF
where
(ax) = 2m
12/2
((x) = / (uu) du.
Thanks to the previous theorem, we can prove the following crucial results.
Theorem Let be a sufficiently large constant. Let (G.).E(.....n-I) be Gaussian O-mean unit variance random walk, and μ R. If μ 2 CT .for some a E (0, 1/2) and C 4log(T),, then setting n [ we have
P
Σ 1 (-0.0) (G,+ + in) - T2a
21 - - 27-1/2
Proof. In the rest the proof, we will assume, for ease notation, that T is such that / an integer, so that n 71/2+a, This is done without loss generality, since substituting n with n + leads to neglighle difference for T sufficiently big. Applying the discretization corollary we have that for every= a E 10
n-1
2log(n)
V2 v T (oo(o)))))
Σ i=
1 x- 00 (G, + ip) S sn 2 BB - Vn
58)
(6)
where μi vn,. Therefore, by assumption,
H4 Vnp 111/2t031/ CT'
At this point, we can apply Theorem to have, for any 17 > O,
P(B: + tpo 7l s 8) = / [ -
1 /1 -T)
VI-r
HOT
moe
(- HOT ( ))
dr
VF
VF
which means that
IP (BB, + too I ml S = 1- f'
KHovi
+ no 1 --)
VI-7
(2)
x
Φ
17
/
HOT
1
une
2ron
-n) -TTT
/T
dr.
Here, we have to consider that
11 2 logín)
colog(T) 2/
1/2-1/4
vn
no 2 14-0//
Moreover, to have the thesis, interested in value of such that T200 corresponding to 1r21/2+a Therefore, in the interval 1/2++ 1], we have
23

---

1. Consider term (3):
1 - MOT
1
1] ター -o 1/240 11/40
1
VF _
T
T 7-1/4-0/2
T-1/440/2
(1/11/4-a/2 140 NoT 1/1//////2)
T-1/Lta/2
Here, since 2lo(()) Vn
2log(T)T~a/2-11 1/4 the part 1/I 2/1//4//2 a/2 is bounded by
2 log(T).
Instead, po 1/4+a/2 2 1/1//4///3--1/taa22222 C.
2. Term (4) is non-negative.
Therefore, for log(T), we have that in the interval [T-1/2+a, 1]
(3) + (4) S & (2)og(T)) T T-1/1+a/2
1
1
2log(7)?
T-1
V22TT
With this inequality, we have
T 1
( ( dr
[B. + 1u0 /n S ST-1/2t0
=
2 JT 1/21a
v1
2 1 T-1
+ luuldd
2T
T-12240
V2T -)
2 1 T-1
+ +looldT
2T
/2 -r)
= 1 - 2 T-1
[22
+
2T
At this point, knownig from the assumptions that n < T, we have μu S VT, which implies
T-1/2
P P + tyo S nl S 1T-1 21
π
Substituting this result into Equation 6 we get, for s = T-1/2+a and n > 11/2+a
T/2
V2
P
(-0.,0 (Gr + ip) < 22a
T/1/2+a log (T2/2+a)
π
2 1- - 27 -7 1/2
The second result is the following Theorem 8. unit variance random walk, and E TR. such that < CT for some E (0,1/2) and C' = (Tg + 2. Then, for any (0,1/2), setting n = | we have
a sufficiently large constant. Lei
be a Gaussian 0-mean
P Σ
1-00,00 (G++ + ip) 22
S 11/2/0
24

---

Proof. In the rest of the proof, we will assume, for ease of notation, that T is such that T (1/2+a an integer, so that n. = 7 This is done without loss of generality, since substituting n with n + 1 leads a negligble difference for T sufficiently large. Applying the discretization corollary [ have that for every s E (0,1) 1)
1 0c.0| (G₁ + ip) I sn SP (EP(BB+tuu - 2 2log(n) vn
v2 Jn log(n)
P
Σ i=
< +
(7)
v
where μi Vnu. Therefore, by assumption,
Po Vnp < - Differently from the previous proof, here cannot directly apply Theorem
2 vn
since 77
Still, we can say that
2 2log(n) P ( B: two 5
2 2log(n) n 2 llg((n)
S
=
Br
tuo
S
P
(B.
tju
1
Vn
v' ji Po and B, -B, (it is not necessary rename since its distribution is symmentric). In this way we can apply Theorem 6 having that the previous probability corresponds to
At this point, set 77 2log(n)
/n/>1 >1 8) 2 f(
6(poV1-7 -T-r
P P[B, + tio S
+ Doo -
(2)
(1)
④
17
POT
1
Moe
2
17
Puu
dr.
VT
(3)
(4)
Here, we have to consider that
11 2 logín))
vn 2 (oo
Ao CT
Moreover, to have the thesis, we are interested in a value of s such that sn = T corresponding to
112/2+a
Here, it is convenient to divide the proof in two cases, depending on the sign of a/2 -
1. Assume (1/4 + a/2 2 0). Then, considering term (3) we have that for r E [1/2. 1]
(3) S Φ For
< V26 (V2n- (V2n-Wo/V2))
Moreover, since term (4) is nonnegative also have
+ (4) < /2 (V2n io/V2)
1 V
(V2n Boo-////22/// /2//2/1
Being 1/4 + a/2 0 > 0 and < 1, the exponent is less than - - C/V22///22 This
means that for C' 2 /o (T) + the full term is bounded by
25

---

(V2log(T)) T
(3) (4) S (v2
VT
V
Substituting this inequality, we get
IP (IB, + tio 1 nl > -1-18 T
1/2ta)
-
2T-11
[(poov ( T)
+ + +200(AOv1-
dr
VT
1-T
v1 7
2T
1
1
V
1-T 1/2
/ 2/ - T)
21"-1
67
(2 + T 1/240 (ua) S
VT
This quantity is of course less than T-8 since 8 E (0,1/2) by assumption
2. Assume (1/4+ a/2 0). In this case, we have, being jóu 2 0, the following inequality
P (| 11o S n/ > s) > 1 s).
This simplified form leads to
(0))
7 1
P P( + tio S <n/> > - S2
dr
1-s v1 VT VT"
(00)
2
(0)
dT
1 v1
dr.
Since in our case T-1/2+a < 1/2, this can be further simplified
1 dr
(IB + tăo S > - 5
# JI-s
VT(1
7)
dr
1-4
V1 -
y=1-r
2 1
dy v8.
4 π
π
V
This leads to
P P -too two S 1/ T T 1/2+a) T-1/2+1)))))
< 1/1//24
π
By assumption, 1/4+o0 a/2 0 < 0 the exponent -1/4+ a/2 < T 1/2+0, Therefore, we have
4 T 1/2+8
P
[B, + tāū S >1 T - <
Thus, we have proved that in both cases
( B, + tÃo 51/>1-5)
T 1/24/
π
26

---

Finally, applying Equation 7 and substituting the value of we get
v2
Σ
1 -x,,0] (G, + ip) T
S --1 T 1/2+0
HT log (T1 (1/2aa)*
which implies
Σ
1 [-0,0] 0f (G; + iip) S T2a
S 377-1/240.
B.2.3
Regret bound
Before the actual proof, are stating simple proposition about the structure of the loggrid, which will ease the next computations.
Proposition Let
LG(1/2,1,7)= = ITT ((TAA)))((((--1111111 >3 [log(T))
Vj = -....., (g...,[[og(TT]1
The following identities hold
LG(112,11TT) can be equivalently defined as
LG(1/2,1,7)= = { {(T 1 auaerry J, Vj O,..., [T))(T))
2. Let Ej the j-th element of LG(//2,1T)), and aj = Tog(T) 1/2-1
log(e,)
1/2. Then aj [[og((T]]
3. The ratio of two consecutive values of lg is frs by
so Trcco E [Ve,2) T 51.
Next, prove the following lemmas, which concern some features of our algorithm.
Lemma 6. For any i, the probability of the event Erio corresponding to i eliminating the another in arm such that their gap is Dis, := Pii - p4 > O, is, at most
P((Fa.)) 6log (1gg())(44og(T)))) 2)7 A_ -1
Proof. Let call:
A Hio 4log(T) +
ANo
At this point, there are two possibilities.
1. Ano I 1/2; in this case, the statement of the lemma is vacuous.
2. An, > T-/22 in this case, by assumption, there are two consecutive lj fj,+1 E L such
that
T1/2 T1/2
Án E
C 4,
This is true due to the fact that that the sequence E, spans from 1/2 to T. By Proposition 9 this can be equivalently expressed by saying that
Ano E
T
( c.l
[
27

---

Let us define the following family of events:
Frio (=) == arm eliminates arm io when both have been pulled times.
The probability of Exi, is bounded by:
P(Eni))) - PP ( Eito(())
=P
(2ml
EEC
[
IP( Ersoo (Lz)).
Here, we have applied the fact that, by design of the algorithm, the arms only be discarded in fair steps for which z E L and then a union bound. Here, we notice that by definition the filtering condition, defining as the fair time-step where both arms have been played & times, this event can be again rewritten as:
(R-(1) > (io( 2 T2a;
I:T fair
where o, log(E (T 2- If we call fr, Ar, the empirical means of arms in, i after 7 pulls of each, the previous event can be interpreted as the time in which the random walk given by the difference of the rewards the two arms stays (-o0, 0j:
(1) > R, (1o)3
Σ T=1
1 tirt 2 Pr.io)
Tim Lor fair
&
1 (-0000
Σ k=1
ri,k
Ti,k
where Ei= fi,k is the cumulative reward of arm i and Ei= F1,k is the cumulative reward of arm 2o. Therefore, we have written this quantity as the time spent by the random walk G, in the interval Ly. (-o,,0), for ly. The drift term for this random walk is given by:
E Frra ri,4] = Mip 14 -
Therefore, apply Theorem for the following choice of parameters,
(a) a = aj [To + 1/2) (Proposition 9. which implics n = [T1/2+a5]] = lj.
(b) 0 = i.+1 We can use this choice since the drift is [[log(T)]
Ait ikg
(4log(T) +
A Eko
22
(1g())+
2T
therefore the assumptions of the theorem are respected.
Applying the theorem, we have:
((EE) S 3 1/2+0 = 3T - 2[62(113_
1/2+
Summing over j. get,
T
U
E()
< Σ ((E))
1/2+ 1++1 211661111
< 3 (TYT
28

---

To conclude, consider that, by definition.
Ej = 1/2 2لo
this means that: so that, in particular,
Diio
(I-Jlonte))) fott
T नाo
Î E5p 2 T'
and
A 2 (4) 2)T
Substituting in the bound we have just found results in,
logg 1/2+ дلليीरग
(lglo)))) + 2y7 1/27 नगकd A
< (log(()(Alog(II) + 1/2' -1
Lemma 7. For any arm E [n], the probubility of the event B corresponding arm E [n] not being eliminated after (8log(T) + 42 -1 pulls if there is an active arm in with Duo Hig μ4 > O is such that:
P(E S 2T 1/2
Proof. Define aj ay as in the previous lemma, so that aj
log(f,) Tog(T)
2. As in the previous lemma,
we define:
No
4log (T) +
and j. E (I.... [log(T)}} such that:
A E T ST 2
Jo41
Here, remember that by definition of the filtering condition, defining 4, as the fair timestep where both arms have been played Ej, times, Eg- is included in the following event:
(R. > (1) > T:20.
r= Fair
As before, this event be interpreted the difference between two random walks being negative, due to the fact that:
Σ >
R+ Σ ( 1 2 (
(-00,0
rig k
rikk
In this formulation, we have written the quantity of interest for the filtering condition after Lj +1 pulls as the time spent by the random walk G, in the interval (-o0, OJ, for 1,... +1. This time, the drift term is:
E Fink - ri,k] = Mio - Pa = Aiio
Therefore, can apply Theorem for a aj., since, by assumption,
As (4 (ogg +
24log(T)
2T F
29

---

This theorem leads to:
+1+
PP ( 1 -
(-00,0) (G,) S +2ash1
r=1
Thm
S 27
27 1/2
(8)
To get the thesis, is sufficient reformulate the critical time-step f,+1 +1 in terms of Arig By definition,
T2/2
T1/2
A iig
TT P
Ej.
J.+11
Therefore, Ano I (8log(T) + +4) T1/2 3,
From this, it immediately follows,
4,+1 S (8log(T) 4)7
concluding the proof.
We are finally able prove our main result about the indeeeedend regret of the algorithm. Theorem 4. In the stochastic bandits with ranking feedback setting, when the noise is Gaussian, Algorithm 2 achieves Rr S 62n (
Proof. Fix sub-optimal arm with corresponding gap A, with respect the optimal and let tb a parameter to be chosen later. Define, for every couple of indices i,, in the event:
E
{
{ Eigio
Pio
pii > +
else,
where the event E is defined d eliminating arm in in some point of the process. The probability that at least one of this events verifies bounded by lemma and union bound with:
(VV) :P
< 3n (og 2)2-1/2
6150
11=15500
their number is most nnn 1)/2. Therefore, under the complementary of V elimination with gap larger than t happens, we are sure that an arm i' with gap (w.r.t. the first arm) less than (n- 1)bb survives until the last: in fact, most n climinations may happen, and all between pair of arms with a difference at most 4/.
As the arm it is active until the last, the probability of event E;- that survives for more than (8 lTg + 4)T A pulls is bounded by Lemma 7 with 1/2 Making the union bound over all possible values of i", which is a random variable, we can say that the probability that any this event happens at most 2(n 1/22 1/2 Summarizing, we have the following bound on (T")::
1. (T) < T if either v verifies, which happens with probability,
3n(n 1) (44 + 2)7 1/2
if Eg- verifies, which happens with probability most 2(n 1)T 1/2
2. (T) S T(8 log (T) + 4)T2/²(A, - (n - 1) otherwise. The comes just from the fact that Aur A (n 1)a. by definition of i*
we take v 2(n-1) it results in,
E(A,((T)) < A,TIPU Eyg (A,P((E))) (,T((1og(T)) + 1/2 (A, - (n 11))))
< A,T (3n(n - log(T) + 1/2 11/2
30

---

+ (,T(8log(T) + 4)42//
(n 1)
I
A 6n(n 1) log(T) -2 2nT
2A,7 log + /)
6n(n 1) log (T) (T) + 2)T 2n 2T(8log(T) + 4//2//2
Being A, (0, 1) and n 2 the previous quantity is bounded by, ([)) < 62m log(T)
The proof is completed by using the the Regret Decomposition Lemma Lattimore and Szepesvari 2017].
C Proof for adversarial setting
Theorem 5. In adversarial bandits with ranking feedback, there exists a constant Y (0, 1) such that no algorithm achieves o(I') regret with respect to the best arm in hindsight with probability greater than 7/-
Proof. This negative result follows from the impossibility to achieve Rr < CT regret by any algorithm, with C properly set constant and probability 1 E, in all three instances reported next. Please notice that, this result implies that even the no-regret property cannot be achieved in the bandit with ranking feedback setting.
Without loss of generality we consider rewards function bounded in [0, 10). Consider three instances, with two arms do; for each and the associated rewards, defined as follows:
1
At
E
3
Instance D:
a1:0
Vt
AA
0
on: 14 E
1A
2
14
E
3
Instance 2
01:0
Vt
4A E
2
4n VI
WEE 2
10
VI E
3
Instance !
01:0
1A
WE 2
0
14
E
where phase made by the first T/4 rounds, phase made by the last 7 rounds and 5 is near to O.
is made by the next T/4 rounds, phase
In phase 1 all the instances have the same ranking feedhack, as the first action gives higher rewards with respect to the second one. To make instance 1 receive Rr I CT, it is necessary:
5 T E[nao] & CT = El E(mno)] > (1 2C/T
2
where Ray is the number of times the first arm has been pulled, and the expected value is taken on the randomization of algorithm. From previous equation we obtain that in all instances:
Rag
> 21 20)T - -
(1-200
II ()T/4
where C 8C', Nap is the number of time the first arm has be pulled in phase and the inequality is computed considering that is played in all the next phases. By reverse Markov inequality:
C-
(1 C.TT1
IP
C
31

---

Setting the probability equal to 9/10 we obtain:
C 10C1
from which follow that with probability 9/10 we have:
Rag
10C
and consequently:
10C
We observe that in the second phase, instances ? and ) have the same feedback. Proceeding as done before, make instance ? receive Rr S CT necessary:
3 T [ & CT = E nnaill
-
1 4
3
4
From previous equation obtain that in instances and :
- C2//4
Yaa
where the inequality is computed considering that a₁ is played in the next phases and C2 = 4C. By reverse Markov inequality, obtain that, with probability 9/10:
(1 - 10C2)7/4
and consequently:
Rao I 10C2T/4
We neglect the 5 value for now, as it can be chosen to be insignificant with respect to the previous computation. Now we focus the third phase, in which instance 2 should play:
0 T T/4 T- (1 C3)T/2,
nai
where C3 = 2C. By reverse Markov inequality, we obtain that, with probability 9/10:
10C3)7/2
and consequently:
Tap I 10CgT/2
Now, we compute the number rounds needed the third instance switch the ranking in the third phase, namely q. Notice that, until this switch, the last two instances receive the same feedback. We compute q in the best-case scenario (that is, when small value is sufficient to allow the switch) that satisfies the constraints previously shown. Precisely, q computed so that the empirical mean of
arm Op is greater then the arm a1 one, given that nlno Formally:
> (1 10C and Doy
> (1 10-20//////
0(1 - 10/4 + 10g
0T/2
q (1 11-110C)Y//
10C (1-
We now show that for proper C' value we can lower bound the right side with $. In particular:
00,107/4 110+++11100000000 100))T//////////
100, 7/4 + (1 10C;)7/4+7/2 > = C < 1/200
which means that, for C < computation. Moreover, notice that gap between 1 and 0C₁ 107/4+(1 -1044(1111001,)11 (/C.)T/44077 allowed us to
can substitute the right side the equation with 1 to simplify the
neglect the computations with 8. Then:
0 10C1)T/44 10g
( 39
20C) T/4
2 1/4
+ (1 10C,)7/4
32

---

To achieve contradiction, sufficient to find C' that q + Mlay T/2: indeed, the previous inequality shows the impossibility to gain enough rewards to make the ranking change and, at the same time, guarantee the minimum rewards to make instance @ no-regret. Given that the ranking switch a necessary condition to make instance ① no-regret, the result of impossibility follows for:
39 (1 - T/4+ +1 (1- (1
4
1640
To conclude the proof, we show that the intersection between the events derived by reverse Markov inequality (namely E; with i E (3) holds with constant probability:
P
n EE
E,
P
U E
iE(3]
2 1
PP()))
sE|3]
3
10
10
where the inequality holds by Union Bound. Substituting all the previous results in the definition of
regret we obtain, with probability T0 = é and C < 1640 RT 2 CT = -I((') which concludes the proof.
D Numerical evaluation
This section presents a numerical evaluation of the algorithms proposed in the paper for the stochastic setrings, namely, DREE and R-LPE. The goal of such a study is to show two crucial results: firstly, the comparison of our algorithms with well-known bandit baseline, and secondly, the need develop distinct algorithms tailored for mnstace-deeendeen and instance- mestace-inddepedeen scenarios.
To establish benchmark for comparison, we consider the EC Caapplore-The--Comeeee which is one of the most popular algorithms among the explorethhen-commit class providing sub- linear regret guarantees. In the following, we evaluate the DREE algorithm with different choices
algorithm,
of the parameter in the function f(t) = log(t)++5; precisely, we choose E {1.0, 1.5, 2.0}. Furthermore, we consider four stochastic instances whose specific parameters are discussed below. In all these instances, we assume the rewards to be drawn from Gaussian random variables with unit variance, i.e., 2² 1, and we let the time horizon be equal T 2 10 Finally, for each algorithm, we evaluate the cumulative regret averaged over 50 runs.
We structure the presentation of the experimental results into two groups. In the first, the instances have a small Amin, while in the second, the instances have a large Amin-
Small values of Amin We focus on two instances with Amin < 0.05. In the first of these two instances, consider = 4 arms, and minimum gap of Amin 0.03. In the second instance, consider n = 6 arms, with Amin = 0.03. The expected values of the rewards of each arm are reported
in Section D.I while the experimental results terms of average cumulative regret are reported in
Figures D 2 We observe that in the first instance (see Figure I all the DREE algorithms exhibits linear regret bound, contirming the strong sensitivity of this family of algorithms on the parameter
Amin in terms of regret bound. In contrast, the R-LPE algorithm exhibits better performances in terms of regret bound, as its theoretical guarantee are independent on the values of Amin- Furthermore, Figure [iooesssswwwss that the DREE algorithms (with 5 E 1.0. 1.5) achieve a better regret bound when the number of arms is increased. Indeed, these regret bounds are comparable to the ones achieved by the R-LPE algorithm. The previous result is reasonable as the presence of A in the regret bound lowers the dependence on the number arms. is worth noticing that all our algorithms outperform the baseline EC.
33

---

3,000
EC
DREE (8 1)
DREE (5 1.5)
Cunulative Regret
DREE 2)
2,000
RLPE
1,000
0.5
1.5
Time
100
Figure 1: Instance with Amin 0.03 and all the gaps small.
-104
1.5
EC
DREE (6 [6=]] 1)
DREF (6 1.5)
Cumulative Regret
DREE (6 2)
1
RLPE
0.5
0
0.5
1.5
Time
-10
Figure 2: Instance with Amin = 0.03 and the other gaps big
Large values of Amin We focus on two instances with Amin 2 0.25. the first instance, we consider 4 arms with a minimum gap of Amin 0.5 among their expected rewards. In the second instance, we instead consider larger number of arms, specifically = 8, with a minimum gap equal Amin 0.25. The expected values the rewards are reported in Section D.I while the
experimental results in terms of average cumulative regret are provided in Figures 3 3 As it clear from both Figures 4 when Amiu is sufficiently large, the DREE algorithms (with achieves better performances with respect both the EC and R-PLE algorithms in terms of cumulative regret. Furthermore, there empirical evidence that small guarantees better performance, which is reasonable according to theory. Indeed, when ó is small, the function [()). which drives the exploration, closer a logarithm. Also, as shown in Corollary [ when Amin large enough, the parameter 8 affects the dimension of C(f, A,) more weakly, which results in a better regret bound.
{1.0, 1.5))
34

---

8,000
EC
DREE 1)
DREE (6 = 1.5)
Cumulative Regret
6,000
DREE 2)
RLPE
4,000
2,000
0.5
1.5
Time
-10
Figure 3: Instance with Amin 0.5.
-10
EC
DREE (6
Cumulative Regret
1.5
DREE (6 1.5)
DREE
RLPE
0.5
0
0.5
1.5
Time
.105
Figure 4: Instance with Amin 0.25.
D.1 Experiments
For the sake of clarity, we report in the followings additional details on the four instances presented Notice that, all the plots present the cumulative regret averaged for 50 runs, with 95% confidence interval. Furthermore:
Figures 32
Instance of Figure time horizon T' 2-500, arms n mean reward vector
μ = [0.9, 1.05, 1.12, 1.15),
unitary variance for each arm, Amin = 0.03;
Instance of Figure 2 time horizon T 10 arms n = 6, mean reward vector
[0.03, 0.07, 0.1, 0.08, 0.97,
unitary variance for each arm, Amin 0.03;
Instance of Figure time horizon 7 2 105, arms n = 4, mean reward vector f [0.05, 0.25. 0.5, 1.0).
35

---

unitary variance for each arm, Amin 0.5;
Instance of Figure time horizon 7' 2 100, arms n = 8, mean reward vector
/ [0.05, 0.05, 0.15, 0.25, 0.5, 0.75,. 1.0),
unitary variance for each arm, min 0.25;
D.2 Detailed explanation of the experiments
this section, we report all the details of the experiments performed in the paper. These are important ensure the truthfullness of the results and the claims based empirical validation.
Training details In the main paper we have presented four experiments, each corresponding to a different environment. Each experiment is performed for fifty random seeds, ad the computation split in 10 parallel processes by the library joblib. The overall computational time for one experiment is around 337.97 .92 seconds, that is roughly five minutes and one half.
Compute As stated, the numerical simulations resulted to be very fast. For this reason, it was not necessary run them server, and used personal computer with the following specifications:
CPU: 11th Gen Intel (R) Core (TM) 17-1165G7 2.80 GHz
RAM: 16,0 GB
Operating system; Windows
System type: 64 bit
Reproducibility Due to the stochastic nature of the bandit problem, all the simulations have been repeated several times. We have performed all the experiments with 50 different random seeds, corresponding precisely to the first 50 natural numbers. The seed influences the generation of the reward by the environment, while all algorithms proposed, being deterministic, are independent the seed.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All we say in the abstract is done in the work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clarifiy that the contribution theoretical, which is the only aspect that can be seen as limitation of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
37

---

Justification:
We leave the complete and formal proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: the appendix we report every detail of the experiments, even the amount of CPU time used.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
38

---

Answer: [Yes]
Justification:
We provide the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee 7/nips.com / casssiinnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
These information are reported in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Every experiment reports, together with the regret curves, shaded indicating the uncertainty.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
39

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
These information are reported in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Experiments are just Python simulation, concerns about real data.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Only thcoretical work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
40

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We did not use data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: We only use standard Python libraries make the experiments.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
41

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Answered in previous questions.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not do any crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We have no study participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "miYmQXviUQ", "reviewer": "Reviewer_hpCM", "summary": "This paper considers a new multi-armed bandit problem where the feedbacks are ranking of the arms. In this problem, the environment gives the feedback on the ranking of the arms based on the previous pulls. The authors first consider the stochastic setting and give the lower bound of the regret for the instance-dependent case. Then, they propose a design of explore-then-commit, called DREE, which is proved to achieve a sublunar regret for instance-dependent case. After discussing the regret tradeoff between instance-dependent and  instance-independent cases, the authors design a phase elimination algorithm (R-LPE) that has a sublunar regret. Then they move beyond the stochastic setting and prove that sub-linear regret cannot be achieved for the adversarial setting.", "questions": "Can the authors give more concrete examples to motivate the setting of bandits with ranking feedbacks?\nCan the authors provide a high-level explanation on why there exists a tradeoff between instance-dependent and instance-independent cases?", "limitations": "The authors do not include a discussion of the limitations. However, a discussion on the limitations of the setting will be helpful.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper has the following strengths.\n+ The paper considers a novel setting of bandits with ranking feedback. The ranking feedback depends on the history of reward,  so it is different from the dueling bandits. \n+ For the stochastic setting, the paper derives the lower regret bound to show the difficulty of the instance-dependent case. \n+ The paper proposes provable algorithms to solve bandits with ranking feedback for both  instance-dependent and instance-independent cases. Adequate analysis is provided to support their points.", "weaknesses": "I am concerned about the following aspects.\nFirst, in the problem setting, the ranking feedback is assumed to be perfect in terms of the ranking of the averaged history rewards. However, human probably not give the perfect ranking since history reward is not easily observed. Thus, it would be better if the authors can discuss the designs with imperfect ranking feedback.\nSecond, the analysis in the adversarial setting does not give any insight. It is obvious that the regret is linear for adversarial bandits with ranking feedback. However, can the authors discuss whether their algorithms are robust enough for adversarial setting? (e.g. provide an analysis of the competitive ratio.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: _\"First, in the problem setting, the ranking feedback is assumed to be perfect in terms of the ranking of the averaged history rewards. (...)\"_\n\nWe completely agree with the Reviewer that the introduction of imperfection/uncertainty/tolerance is of paramount importance as it would allow to capture better the actual human behavior, and this refinement is exactly the next step of our agenda. On the other side, no imperfection can be introduced before a complete study of the perfect model, and, as we show in our paper, the study of the exact model is not straightforward. Furthermore, we observe that introducing a perturbation in the ranking observed by the learner may hinder the possibility of designing a no-regret algorithm. Thus, exploring scenarios where the learner receives corrupted ranking feedback and still manages to design a no-regret algorithm is both a fascinating and challenging research direction. Nonetheless, it requires a different approach from the one presented in our work and is something we plan to investigate in the future.\n\nQ: _Second, the analysis in the adversarial setting does not give any insight. (...)_\n\nWe believe that the impossibility result for the adversarial setting represents a first yet fundamental step toward a full understanding of the problem when the rewards are adversarially chosen. We are disappointed that the Reviewer found our result trivial, as it is uncommon in the online learning literature to achieve a positive result in the stochastic setting that cannot be extended to the adversarial setting. This feature is peculiar to the bandits with ranking feedback model, as even related settings, such as dueling bandits, do not exhibit this kind of impossibility result. Finally, we agree with the Reviewer that studying the competitive ratio of algorithms developed for adversarial settings is an interesting research direction, and we plan to pursue it in the future.\n\nQ: _Can the authors give more concrete examples to motivate the setting of bandits with ranking feedbacks? Can the authors provide a high-level explanation on why there exists a tradeoff between instance-dependent and instance-independent cases?_\n\nWe thank the Reviewer for the question. In the following, we provide a real-world example of a possible application of our setting. In pay-per-click online advertising (the total spent is of the order of several billion USD per year), large platforms optimize advertisers' campaigns. Specifically, these platforms observe the number of clicks of each single campaign, but to allocate the budget most effectively (using a knapsack-style approach), they need to know the revenue of the individual campaigns. Obviously, the platforms cannot observe the revenue, which is private information of the advertiser. On the other hand, advertisers do not want to communicate this private information to the platforms, and, for this reason, the platforms limit themselves to maximizing the number of clicks. However, this kind of optimization leads to very approximate solutions compared to considering the revenue as well. The use of bandits with ranking feedback  in this context would circumvent this problem. In particular, advertisers would be asked for feedback on the ranking of advertising campaigns, avoiding the need to ask for revenue information.\n\nQ: _Can the authors provide a high-level explanation on why there exists a tradeoff between instance-dependent and instance-independent cases?_\n\nUsually, in multi-armed bandits, an instance-independent regret bound can be derived from an instance-dependent one. Indeed, if the instance-dependent regret bound depends on the suboptimality gaps as $ R_T \\propto {\\log(T)}/{\\Delta} $, it is possible to take the worst possible value of $ \\Delta $ and achieve the desired instance-independent regret bound. Unfortunately, in our case, the instance-dependent regret bound cannot be written in this form as a consequence of the feedback characterizing our setting. Indeed, we can only observe switches in the ranking, which do not reflect the actual differences in the expected rewards of the arms. Thus, the only way to “explore” in our setting is by pulling arms that could be **highly** sub-optimal. For this reason, achieving an asymptotic regret bound that is close to logarithmic requires exploiting the arm being ranked first several times, thus suffering a particularly unpleasant dependence on the suboptimality gaps $ \\Delta_i $ (see, for example, Corollary 3).  Finally, we remark that the formalization of this reasoning is presented in Theorem 4 (Instance Dependent/Independent Trade-off).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HpzEpY3Sv4", "reviewer": "Reviewer_iUMy", "summary": "The paper introduces a variant of the multi-armed bandit problem called \"bandits with ranking feedback,\" where the feedback ranks the arms based on historical data without showing precise numerical differences. This approach is particularly useful in scenarios where exact measurement of values is impractical, such as with human preferences or confidential data. The main contributions of the study include developing no-regret algorithms that operate under both stochastic and adversarial conditions for this model. The findings indicate that achieving logarithmic regret is impossible with ranking feedback in the stochastic setting, and no algorithm can achieve sublinear regret in the adversarial setting. The paper proposes two algorithms: DREE, which achieves superlogarithmic regret in stochastic instances, and R-LPE, which manages a regret of O(\\sqrt{T})in stochastic instance-independent scenarios. These innovations significantly enhance the understanding and implementation of bandit algorithms in complex feedback environments.", "questions": "1) Originality: Is the concept of \"bandits with ranking feedback\" truly novel?\n2) Experimental Validation: The paper would benefit from more experimental validation. While the theoretical aspects are well-developed, further empirical testing in diverse conditions could strengthen the validation of the algorithms. Tests with non-Gaussian noise and in real-world settings would be particularly insightful.\n3) Algorithm Complexity: The discussion on the computational complexity and practical scalability of the introduced algorithms is limited. More detailed analysis in this area could provide better insights into their applicability in real-world scenarios.\n4) Adversarial Setting Analysis: It is mentioned that no algorithm can achieve sublinear regret in adversarial settings, but more detailed explanations or suggestions for alternative approaches to handle such conditions would be beneficial.\n5) Impact of Dependence on Parameters: The algorithms appear to heavily rely on the correct setting of specific parameters, such as the time horizon T. Discussing the sensitivity of the algorithms to these parameters and providing strategies for effective parameter tuning would aid their practical application.\n6) Extension to Other Models: Could the principles of ranking feedback be applied to other types of bandit problems, such as contextual bandits or those with structured action spaces? This extension could broaden the applicability of the research findings.", "limitations": "Experimental Scope: The authors could extend the experimental validation to include real-world datasets or scenarios to better demonstrate the practicality of the algorithms under different environmental conditions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Quality: The theoretical contributions are robust, including the proof that no algorithm can achieve logarithmic regret in the stochastic setting with ranking feedback, and no sublinear regret is achievable in the adversarial setting\nClarity: The paper is well-structured, with clear delineation of problem settings, algorithmic approaches, and theoretical analyses.", "weaknesses": "1) Originality: Is the concept of \"bandits with ranking feedback\" truly novel?\n2) Experimental Validation: The paper would benefit from more experimental validation. While the theoretical aspects are well-developed, further empirical testing in diverse conditions could strengthen the validation of the algorithms. Tests with non-Gaussian noise and in real-world settings would be particularly insightful.\n3) Algorithm Complexity: The discussion on the computational complexity and practical scalability of the introduced algorithms is limited. More detailed analysis in this area could provide better insights into their applicability in real-world scenarios.\n4) Adversarial Setting Analysis: It is mentioned that no algorithm can achieve sublinear regret in adversarial settings, but more detailed explanations or suggestions for alternative approaches to handle such conditions would be beneficial.\n5) Impact of Dependence on Parameters: The algorithms appear to heavily rely on the correct setting of specific parameters, such as the time horizon T. Discussing the sensitivity of the algorithms to these parameters and providing strategies for effective parameter tuning would aid their practical application.\n6) Extension to Other Models: Could the principles of ranking feedback be applied to other types of bandit problems, such as contextual bandits or those with structured action spaces? This extension could broaden the applicability of the research findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: _Originality: Is the concept of \"bandits with ranking feedback\" truly novel?_\n\nTo the best of our knowledge, the \"bandits with ranking feedback\" model introduced in our paper represents a new bandit setting. Although our setting shares similarities with dueling bandits, the two settings are substantially different, as we discuss in the related works section in our paper.\n\nQ: _Experimental Validation._\n\nFollowing the suggestion of the reviewer, we enriched our experimental evaluation, and the new results we obtained can be found in the attached PDF. In particular, we investigated the following:\n\n- **Non Gaussian noise.** In the attached PDF, we ran the same experiments of the paper with uniformly distributed noise. We observe that this leads to similar results. However, the variance of the regret curves is slightly increased, but the mean remains nearly the same.\n      \n-  **Measuring the computational effort required by the algorithms.** While the running time of DREE is clearly linear in $T$, the running time of R-LPE is less straightforward to compute. However, in the additional experiments, we empirically observed that the running time of the R-LPE algorithm is also linear in $T$.   \n\n- **Robustness of R-LPE to misspecification of $T$.**  Even if the algorithm strongly relies on knowledge of the time horizon, we have shown that providing the algorithm with a time horizon larger than the actual one is not particularly harmful. On the other hand, as is usual in bandit algorithms, setting the time horizon $T'$ smaller than the actual time horizon $T$ results in linear growth of the regret after $T'$ rounds.\n\nQ: _Algorithm Complexity._\n\nWe thank the Reviewer for underlining this important aspect, which we did not mention in the paper due to space constraints. Both our algorithms are very computationally efficient. In the experiment, we have empirically demonstrated this fact, and we will show it theoretically in this rebuttal. Specifically, the **DREE** algorithm requires either pulling the first-ranked arm (in most rounds) or an arm in a lower position of the ranking according to a deterministic schedule. Therefore, it does not require computing confidence regions, resulting in a running time of just $\\mathcal{O}(nT)$. The second algorithm, **R-LPE**, is more complex compared to DREE, as it requires updating a set of active arms $S$ at certain rounds. Fortunately, this set is only updated during the time steps that belong to the loggrid, and thus it is updated a logarithmic number of times. Therefore, even though there is a summation over $t$ in the definition of the set $S$, the computational complexity of R-LPE is not quadratic in $T$, but rather of order $\\mathcal{O}(T+nT\\log(T))$. The first term in the latter expression corresponds to the \"usual\" rounds, where we simply follow a round-robin strategy. The second term corresponds to the product of $\\log(T)$ (the number of rounds in the loggrid), $T/n$ (the number of \"fair\" rounds), and $n^2$ (the number of possible comparisons between pairs of arms).\n\nQ: _Adversarial Setting Analysis._\n\nWe thank the Reviewer for the question.     \n    **On the results explanation.** To derive our adversarial lower bound we considered three instances. The instances are similar in terms of rewards for the first two phases but differ significantly in the third one. Thus, we show that, if the learner receives the same ranking when playing in two instances with different best arms in hindsight, it is impossible to achieve low regret in both scenarios.\n\n**On possible alternative approaches.** We agree with the reviewer that, to handle such cases, different metrics could be taken into consideration. For instance, it would be possible to study algorithms achieving sufficiently \"good\" competitive ratios, namely, those that are no-regret with respect to a large fraction of the optimum. We leave the aforementioned research directions as interesting future work.\n\nQ: _Impact of Dependence on Parameters._\n\nThe R-LPE algorithm requires knowledge of the time horizon $T$, while the DREE algorithm is an anytime algorithm. In the attached PDF we added an experiment measuring the empirical impact of using a misspecified value of the time horizon $T$. We would like to emphasize that the R-LPE algorithm has no additional hyper-parameters, since the quantities that characterize it, such as the loggrid and the parameter $\\alpha$, can be computed as a function of $T$. Additionally, while the definition of $\\alpha$ at Line 8 could be tuned experimentally, doing so may prevent us from achieving the desired regret guarantees of $\\mathcal{O}(\\sqrt{T})$, which is the primary goal of our paper.\n\nQ: _Extension to Other Models_\n\nExtending our results to settings where the action space is no longer discrete is highly non-trivial, as it would first require rethinking the concept of ranking feedback. However, if we consider linear (or possibly contextual) bandits with finitely many arms, it would be interesting to see if such a linear structure allows for a better dependence on the number of arms (possibly sublinear as in [1]) in the instance indepenednt regret bound. Nonetheless, it should be noted that, from a technical perspective, combining linear settings with ranking ones would require merging the argument based on optimal design for the least squares estimator [2] or self-normalized processes [3] with our bound, which is instead based on Levy's arcsin law.\n    \n[1] Tor Lattimore, Csaba Szepesvari, Gellert Weisz. Learning with Good Feature Representations in Bandits and in RL with a Generative Model.   \n \n[2] Kiefer and J. Wolfowitz. The equivalence of two extremum problems.\n\n[3] Yasin Abbasi-yadkori, Dávid Pál, Csaba Szepesvári, Improved Algorithms for Linear Stochastic Bandits", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "elJhSWpX2d", "reviewer": "Reviewer_KUAx", "summary": "The paper studies the setting where, every time an arm is pulled, a principal gets to observe a reward, but the player only gets to observe the order that emerges from the accumulated rewards so far. The authors study both the adversarial and stochastic, and both the instance-dependent and instance-independent regrets. They provide tight results for all cases, with the only exception being the stochastic instance-independent case, where they provide tight bounds only for the Gaussian model.", "questions": "No questions", "limitations": "No limitations", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is well-written. Intuitive explanations of the results are provided along with a nice description of the algorithms.\n\n- The model is super interesting in my opinion, at least from a mathematical point of view, and quite novel.\n-  The authors provide a very complete picture. They study both the adversarial and non-adversarial settings, as well as instance-dependent and instance-independent regrets.\n\n- Overall, I really enjoyed reading the paper", "weaknesses": "- I miss some more realistic motivation for the model and some concrete applications. As I said, from a theoretical point of view, the model is very interesting, but when I tried myself, I could not come up with a clear application.\n\n- For the general stochastic and instance-independent case, the only guarantee that is provided is that of the EC algorithm which can be easily applied in this setting\n\n- R-LPE algorithm needs to know T", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: _I miss some more realistic motivation for the model and some concrete applications. As I said, from a theoretical point of view, the model is very interesting, but when I tried myself, I could not come up with a clear application._\n\nWe thank the Reviewer for the question. In the following, we provide a real-world example of a possible application of our setting. In pay-per-click online advertising (the total spent is of the order of several billion USD per year), large platforms optimize advertisers' campaigns. Specifically, these platforms observe the number of clicks of each single campaign, but to allocate the budget most effectively (using a knapsack-style approach), they need to know the revenue of the individual campaigns. Obviously, the platforms cannot observe the revenue, which is private information of the advertiser. On the other hand, advertisers do not want to communicate this private information to the platforms, and, for this reason, the platforms limit themselves to maximizing the number of clicks. However, this kind of optimization leads to very approximate solutions compared to considering the revenue as well. The use of bandits with ranking feedback  in this context would circumvent this problem. In particular, advertisers would be asked for feedback on the ranking of advertising campaigns, avoiding the need to ask for revenue information.\n\nQ: _For the general stochastic and instance-independent case, the only guarantee that is provided is that of the EC algorithm which can be easily applied in this setting_\n\nWe thank the Reviewer for the comment but we believe there is a potential misunderstanding on our results. Indeed, the **R-LPE algorithm is specifically tailored for the instance independent case** while it achieves a subotptimal instance dependent regret bound. Finally, please notice that R-LPE instance independent regret guarantees are far better than those of the standard Explore and Commit algorithm which are of the order $\\mathcal{O}(T^{2/3})$.\n\nQ: _R-LPE algorithm needs to know $T$_\n\nThe reviewer is correct. In our settings, due to the specific nature of the feedback, we cannot employ the well-known \"doubling trick\" to relax the knowledge of the time horizon $T$. We leave it as an interesting open problem to determine whether the requirements on the knowledge of time horizon $T$ can be relaxed.  For what concerns the empirical validation, in the attached PDF we added an experiment measuring the impact of using a misspecified value for $T$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "InUqc9wg0W", "reviewer": "Reviewer_cfwJ", "summary": "The paper studies the multi-armed bandit problem when the learner only observes the ranking of the average cumulative reward of all the arms. This is a strictly worse information environment than the standard setting. The paper proposes algorithms that still attains the instance independent and dependent regret rates that match the stochastic multi-armed bandit. It also shows that no algorithms can do better in both the instance-dependent and -independent settings simultaneously.", "questions": "- In the proof of Theorem 4, the policy doesn't seem to depend on $T$. For example, in line 534, the probability of $\\pi(pull 2|E_\\tau)$ is a function of $\\tau$ only. My understanding that the algorithm is allowed to depend on $T$. Does the proof work for this case?\n- I think the title misleads the readers. Ranking feedback sounds like receiving the order of the rewards in the current round. It is up to the authors, but a more informative title would help the paper.", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The setting studied in the paper is novel. I have not read similar setups in the literature before. Although the practical motivation can be strengthened, I think it is an interesting problem to study. One potential application could be some kind of tounament in which only the ranking is observed. \n- Although the information is a lot less than the standard MAB, the authors show that the optimal regret in both cases (instance-dependent and -independent) can still be achieved. This is surprising. The tools used are standard probabily theory, which is a plus to me. \n- The trade-off between instance-independent and -dependent regret is interesting and surprising. This is not the case in standard MAB and I haven't seen similar results before.", "weaknesses": "- It is probably out of the scope of this paper. But the current dependence on $n$, the number of arms, is quite far from optimal ($n^4$) in the instance dependent case. I wonder if the authors have thought about better designs to obtain linear or sublinear dependence. Some discussion would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: _It is probably out of the scope of this paper. But the current dependence on $n$, the number of arms, is quite far from optimal in the instance dependent case. I wonder if the authors have thought about better designs to obtain linear or sublinear dependence. Some discussion would be helpful._\n\nThe fact that our instance-independent bound in Theorem 8 scales with $ n^4 $ is one of the most interesting open points of this paper. The reason for this unusual order may be found in the fact that the ranking setting prevents us from relying on standard concentration properties. Therefore, no \"high-probability\" estimates can be performed, and each time in the analysis we perform a \"union bound\" over the space of arms, this results in an additional $ n$ factor in the regret bound. In contrast, when using a high-probability estimate, the same union bound would result in an additional $\\log(n)$ term, which is negligible. Furthermore, given the ranking feedback setting, it appears natural that at least two union bounds over the space of arms must be performed in order to compare any possible pair of arms. Therefore, we **believe** that a super-linear dependence on $ n $ in the instance-independent regret bound cannot be avoided, but we still **conjecture** that such dependence may be moved to lower order terms.\n\n_Q: In the proof of Theorem 4, the policy doesn't seem to depend on $T$. For example, in line 534, the probability of $\\pi(pull|E)$ is a function of $\\tau$ only. My understanding that the algorithm is allowed to depend on $T$. Does the proof work for this case?_\n\nThe point made by the Reviewer is correct, and very subtle. In fact, the current lower bound only applies to any-time policies, independent of the time horizon. Fortunately, modifying the proof to generalize the result to the case where the policy can depend on $T$ is not difficult.\nWe start the proof by contradiction, assuming that both\n    $$R_T(\\pi_T)\\le C(\\Delta)T^\\alpha \\qquad R_T(\\pi_T)\\le T^\\beta$$\n\nhold for any $T$, given a sequence of policies $\\pi_T$. The \"hard instances\" are the same, both with two arms and small $\\Delta$, and also the event $E_t$ is defined as in the paper. From the assumption, it follows that\n\n$$C(\\Delta= 1)T^\\alpha = C( 1)T^\\alpha \\ge \\mathbb E[Z_2(T)|E_T],$$\n\notherwise the sequence of policies would suffer regret more than $C( 1)T^\\alpha$ in case arm one gives always $1$ and the other always $0$. From this equation, it is easy to complete the proof as usual. \n\nIn the original proof, the only step requiring the policy to be any-time was the limit\n$$\\forall \\eta>0,\\ \\limsup_{t\\to \\infty}\\frac{\\sum_{\\tau=1}^t \\pi(\\text{pull 2}|E_\\tau)}{t^{\\alpha+\\eta}}=0,$$\n\nwhich is not necessary and can be easily avoided working by contradiction. We will put this improved proof in the final version of the paper, we remain at the Reviewer disposal in case they need further clarification on how to change this proof.\n\n_Q: I think the title misleads the readers. Ranking feedback sounds like receiving the order of the rewards in the current round. It is up to the authors, but a more informative title would help the paper._\n\nWe thank the Reviewer for the interesting observation. Nevertheless, we believe the feedback structure the Reviewer is describing is somewhat referred to as \"Dueling bandits\", or their generalization. Thus, we are open to any suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

in the attached PDF, we provide additional experiments.

The authors.

Meta-review: The reviewers overall favor the paper's contributions, among which are the introduction of a bandit model with a new form of rank-ordered feedback, and a comprehensive study of attainable performance in this model. I must add, however, that there were also concerns raised almost unanimously about the practical relevance of this setting in modeling applications. The author(s) responded to this by providing a setting involving pay per click online advertising optimization where an agent must optimize for ad displays without knowing (potentially sensitive and private) revenue information per ad. This appears to have been satisfactorily accepted as a motivating example by the reviewers. Noting that the study of this new model can hold potential and inform decision making algorithms for future purposes, I recommend acceptance of the paper.
Decision: Accept (poster)