Title: Contextual Bandits and Imitation Learning with Preference-Based Active Queries
Venue: NeurIPS 2023
Accepted: yes
Keywords: Contextual Bandit

Imitation Learning

Learning from Expert Feedback

Theory
Abstract: We consider the problem of contextual bandits and imitation learning, where the learner lacks direct knowledge of the executed action's reward. Instead, the learner can actively request the expert at each round to compare two actions and receive noisy preference feedback. The learner's objective is two-fold: to minimize regret associated with the executed actions, while simultaneously, minimizing the number of comparison queries made to the expert. In this paper, we assume that the learner has access to a function class that can represent the expert's preference model under appropriate link functions and present an algorithm that leverages an online regression oracle with respect to this function class. For the contextual bandit setting, our algorithm achieves a regret bound that combines the best of both worlds, scaling as $O(\min\\{\sqrt{T}, d/\Delta\\})$, where $T$ represents the number of interactions, $d$ represents the eluder dimension of the function class, and $\Delta$ represents the minimum preference of the optimal action over any suboptimal action under all contexts. Our algorithm does not require the knowledge of $\Delta$, and the obtained regret bound is comparable to what can be achieved in the standard contextual bandits setting where the learner observes reward signals at each round. Additionally, our algorithm makes only $O(\min\\{T, d^2/\Delta^2\\})$ queries to the expert. We then extend our algorithm to the imitation learning setting, where the agent engages with an unknown environment in episodes of length $H$, and provide similar guarantees regarding regret and query complexity. Interestingly, with preference-based feedback, our imitation learning algorithm can learn a policy outperforming a sub-optimal expert, matching the result from interactive imitation learning algorithms [Ross and Bagnell, 2014] that require access to the expert's actions and also reward signals.

Contextual Bandits and Imitation Learning with Preference-Based Active Queries

Abstract

We consider the problem of contextual bandits and imitation learning, where the learner lacks direct knowledge of the executed action's reward. Instead, the learner can actively request the expert at each round to compare two actions and receive noisy preference feedback. The learner's objective is two-fold: to minimize regret associated with the executed actions, while simultaneously, minimizing the number of comparison queries made to the expert. In this paper, we assume that the learner has access to a function class that can represent the expert's preference model under appropriate link functions and present an algorithm that leverages an online regression oracle with respect to this function class. For the contextual bandit setting, our algorithm achieves a regret bound that combines the best of both worlds, scaling as O (min { √ T, d/ ∆ } ) , where T represents the number of interactions, d represents the eluder dimension of the function class, and ∆ represents the minimum preference of the optimal action over any suboptimal action under all contexts. Our algorithm does not require the knowledge of ∆ , and the obtained regret bound is comparable to what can be achieved in the standard contextual bandits setting where the learner observes reward signals at each round. Additionally, our algorithm makes only O (min { T, d 2 / ∆ 2 } ) queries to the expert. We then extend our algorithm to the imitation learning setting, where the agent engages with an unknown environment in episodes of length H , and provide similar guarantees regarding regret and query complexity. Interestingly, with preference-based feedback, our imitation learning algorithm can learn a policy outperforming a sub-optimal expert, matching the result from interactive imitation learning algorithms [Ross and Bagnell, 2014] that require access to the expert's actions and also reward signals.

1 Introduction

Human feedback for training machine learning models has been widely used in scenarios including robotics [Ross et al., 2011, 2013, Jain et al., 2015, Laskey et al., 2016, Christiano et al., 2017] and natural language processing [Stiennon et al., 2020, Ouyang et al., 2022]. By integrating human feedback into the training process, these prior works provide techniques to align machine-learning models with human intention and enable high-quality human-machine interaction (e.g., ChatGPT).

Existing methods generally leverage two types of human feedback. The first is the action from human experts, which is the dominant feedback mode in the literature of imitation learning and learning from demonstrations [Abbeel and Ng, 2004, Ziebart et al., 2008, Daumé et al., 2009, Ross et al., 2011, Ross and Bagnell, 2014, Sun et al., 2017, Osa et al., 2018, Li et al., 2023]. The second type of feedback is preference-based feedback, which involves comparing pairs of actions. In this approach, the expert provides feedback by indicating their preference between two options selected by the learner. While both types of feedback have their applications, our focus in this work is on preference-based feedback, which is particularly suitable for scenarios where it is challenging for human experts to recommend the exact optimal action while making pairwise comparisons is much easier.

Learning via preference-based feedback has been extensively studied, particularly in the field of dueling bandits [Yue and Joachims, 2011, Yue et al., 2012, Zoghi et al., 2014, Ailon et al., 2014, Komiyama et al., 2015, Wu and Liu, 2016, Saha and Gaillard, 2021, Bengs et al., 2021, Saha and Gaillard, 2022] and contextual dueling bandits [Dudík et al., 2015, Saha, 2021, Saha and Krishnamurthy, 2022, Bengs et al., 2022, Wu et al., 2023]. Different from the standard bandit setting, the learner proposes two actions in dueling bandits and only gets noisy preference feedback from the human expert. Follow-up works extend the preference-based learning model from the one-step bandit setting to the multi-step decision-making (e.g., imitation learning and reinforcement learning) setting [Chu and Ghahramani, 2005, Sadigh et al., 2017, Christiano et al., 2017, Lee et al., 2021b, Chen et al., 2022, Saha et al., 2023]. These studies mainly focus on learning a high-quality policy from human feedback, without concerning the question of active query in order to minimize the query complexity.

However, query complexity is an important metric to optimize when learning from human feedback, as human feedback is expensive to collect [Lightman et al., 2023]. For instance, InstructGPT [Ouyang et al., 2022] is trained only on around 30K pieces of human feedback, which is significantly fewer than the internet-scale dataset used for pre-training the base model GPT3, indicating the challenge of scaling up the size of human feedback datasets. In other areas, such as robotics, learning from human feedback is also not easy, and prior studies (e.g., Cohn et al. [2011], Zhang et al. [2022], Myers et al. [2023]) have explored this issue from various perspectives. Ross et al. [2013], Laskey et al. [2016] pointed out that querying human feedback in the learning loop is challenging, and extensively querying for feedback puts too much burden on the human experts.

In this work, we design principled algorithms that learn from preference-based feedback while at the same time minimizing query complexity under the settings of contextual bandits [Auer et al., 2002] and imitation learning [Ross et al., 2011]. Our main contributions can be summarized as follows.

In the contextual dueling bandits setting, the stochastic preference feedback is generated based on some preference matrix [Saha and Krishnamurthy, 2022]. We propose an algorithm (named AURORA - in short of Active preference qUeRy fOR contextual bAndits ) that achieves a best-ofboth-worlds regret bound (i.e., achieving the minimum of the worst-case regret and an instance dependent regret), while at the same providing an instance-dependent query complexity bound. For benign instances with small eluder dimension and large gap, our regret and query complexity bounds both scale with ln( T ) where T is the total number of interactions in contextual bandits.

In imitation learning, the stochastic preference feedback is generated based on the reward-to-go of the expert's policy (e.g., the expert prefers actions that lead to higher reward-to-go). We propose an algorithm named AURORAE, in short of Active preference qUeRy fOR imitAtion lEarning , which instantiates H instances of AURORA, one per each time step for the finite horizon Markov Decision Process (MDP), where H is the horizon. By leveraging preference-based feedback, we show that, interestingly, our algorithm can learn to outperform the expert when the expert is suboptimal. Such a result is beyond the scope of the classic imitation learning algorithm DAGGER and previously can only be achieved by algorithms like AGGREVATE(D) [Ross and Bagnell, 2014, Sun et al., 2017, Cheng and Boots, 2018] and LOLS [Chang et al., 2015] which require direct access to expert's actions and reward signal - a much stronger feedback mode than ours.

To the best of our knowledge, for both contextual bandit and imitation learning with preference-based feedback, our algorithms are the first to achieve best-of-both-worlds regret bounds via active querying.

1.1 Related works

Selective Sampling. Numerous studies have been conducted on selective sampling across various settings [Cesa-Bianchi et al., 2005, Dekel et al., 2012, Agarwal, 2013, Hanneke and Yang, 2015,

2021, Zhu and Nowak, 2022], with the work of Sekhari et al. [2023] being closest to ours. Sekhari et al. [2023] presented a suite of provably efficient algorithms that are applicable to settings including contextual bandits and imitation learning. The primary distinction between our setting and the prior works lies in the feedback modality-we assume preference-based feedback, whereas they assume direct label feedback or reward signals.

Contextual bandits with preference feedback. Dudík et al. [2015] is the first to consider contextual dueling bandits, and one of their algorithms achieves the optimal regret rate. Saha and Krishnamurthy [2022] studied contextual dueling bandits using a value function class and proposed an algorithm based on a reduction to online regression, which also achieves an optimal worst-case regret bound. In this paper, we mainly follow the setting of the latter and make notable improvements in two aspects: (1) in addition to the O ( √ AT ) optimal regret rate where A is the number of actions and T is the number of interaction rounds, we established an instance-dependent regret upper bound that can be significantly smaller when the bandit exhibits a favorable structure; (2) our algorithm has an instance-dependent upper bound on the number of queries, and thus when the underlying instance is well behaved (has small eluder dimension and large gap), we will make significantly fewer queries.

Another related work is Saha and Gaillard [2022] which achieves the best-of-both-worlds regret for non-contextual dueling bandits. Our setting is more general due to the context and general function approximation, which enables us to leverage function classes beyond linear and tabular cases.

RL with preference feedback. RL with preference feedback has been widely employed in recent advancements in AI [Ouyang et al., 2022, OpenAI, 2023]. According to Wirth et al. [2017], there are generally three types of preference feedback: action preferences [Fürnkranz et al., 2012], state preferences [Wirth and Fürnkranz, 2014], and trajectory preferences [Busa-Fekete et al., 2014, Novoseller et al., 2020, Xu et al., 2020, Lee et al., 2021a, Chen et al., 2022, Saha et al., 2023, Pacchiano et al., 2021, Biyik and Sadigh, 2018, Taranovic et al., 2022, Sadigh et al., 2017]. We focus on the action preference with the goal of achieving tight regret bounds and query complexities.

The concurrent work from Zhan et al. [2023] investigates the experimental design in both the trajectories-based and action-based preference settings, for which they decouple the process of collecting trajectories from querying for human feedback. Their action-based setting is the same as ours, but they mainly focus on linear parameterization, while our approach is a reduction to online regression and can leverage general function approximation beyond linear function classes.

Imitation learning. In imitation learning, two common feedback modalities are typically considered: expert demonstration and preference. The former involves directly acquiring expert actions [Ross et al., 2011, Ross and Bagnell, 2014, Sun et al., 2017, Chang et al., 2015, Sekhari et al., 2023], while the latter focuses on obtaining preferences between selected options [Chu and Ghahramani, 2005, Lee et al., 2021b, Zhu et al., 2023]. Brown et al. [2019, 2020] leveraged both demonstrations and preference-based information and empirically showed that their algorithm can learn to outperform experts. Our imitation learning setting belongs to the second category, and we established bounds on the regret and the query complexity for our algorithm. We show that our algorithm can learn a policy that can provably outperform the expert (when it is suboptimal for the underlying environment).

2 Preliminaries

2.1 Contextual Bandits with Preference-Based Feedback

In this section, we introduce the contextual dueling bandits setting. For notation, we denote [ N ] as the integer set { 1 , . . . , N } and denote ∆( S ) as the set of distributions over a set S .

We assume a context set X and an action space A = [ A ] . At each round t ∈ [ T ] , a context x t is drawn adversarially , and the learner's task is to pick a pair of actions ( a t , b t ) ∈ A × A and then decide whether to make a query to the expert. If making a query, a noisy feedback y t ∈ {-1 , 1 } is revealed to the learner regarding whether a t or b t is better. We assume that the expert relies on a preference function f ⋆ : X × A × A → [ -1 , 1] to samples its preference feedback y t :

where ϕ ( d ) : [ -1 , 1] → [0 , 1] is the link function, which satisfies ϕ ( d ) + ϕ ( -d ) = 1 for any d . If the learner does not make a query, it will not receive any feedback for the selected actions a t and b t . Let Z t ∈ { 0 , 1 } indicate whether the learner makes a query at round t .

We assume that the learner has access to a function class F ⊆ X × A × A → [ -1 , 1] that realizes f ⋆ . Furthermore, we assume that f ⋆ , as well as the functions in F , is transitive and anti-symmetric.

Assumption 1. We assume f ⋆ ∈ F and any functions f ∈ F satisfies the following two properties: (1) transitivity: for any x ∈ X and a, b, c ∈ A , if f ( x, a, b ) > 0 and f ( x, b, c ) > 0 , then we must have f ( x, a, c ) > 0 ; (2) anti-symmetry: f ( x, a, b ) = -f ( x, b, a ) for any x ∈ X and any a, b ∈ A .

We provide an example below for which Assumption 1 is satisfied.

Example 1. Assume there exists a function r ⋆ : X × A → [0 , 1] such that f ⋆ ( x, a, b ) = r ⋆ ( x, a ) -r ⋆ ( x, b ) for any x ∈ X and a, b ∈ A . Typically, such a function r ⋆ represents the 'reward function' of the contextual bandit. In such a scenario, we can first parameterize a reward class R ⊆ X × A → [0 , 1] and define F = { f : f ( x, a, b ) = r ( x, a ) -r ( x, b ) , r ∈ R} . Moreover, it is common to have ϕ ( d ) := 1 / (1 + exp( -d )) in this setting, which recovers the Bradley-Terry-Luce (BTL) model [Bradley and Terry, 1952] - a commonly used model in practice for learning reward models [Christiano et al., 2017].

Assumption 1 ensures the existence of an optimal arm, as stated below.

Lemma 1. Under Assumption 1, for any function f ∈ F and any context x ∈ X , there exists an arm a ∈ A such that f ( x, a, b ) ≥ 0 for any arm b ∈ A . We denote this best arm by π f ( x ) := a . 2

The learner's goal is to minimize the regret while also minimizing the number of queries, defined as:

It is worth noting that when f ⋆ is the difference in rewards (as in Example 1), the regret defined above reduces to the standard regret of a contextual bandit. We also remark that our feedback model generalizes that of Saha and Krishnamurthy [2022] in that we assume an additional link function ϕ , while they assume the feedback is sampled from Pr( y = 1 | x, a, b ) = ( P t [ a t , b t ] + 1) / 2 where P t is a preference matrix. Their loss function is captured in our setting (Example 2). However, Saha and Krishnamurthy [2022] do not assume transitivity.

2.2 Imitation Learning with Preference-Based Feedback

In our imitation learning setup, we consider that the learner operates in a finite-horizon Markov decision process (MDP), which is a tuple M ( X , A , r, P, H ) where X is the state space, A is the action space, P is the transition kernel, r : X ×A → [0 , 1] is the reward function, and H is the length of each episode. The interaction between the learner and the environment proceeds as follows: at each episode t ∈ [ T ] , the learner receives an initial state x t, 0 which could be chosen adversarially. Then, the learner interacts with the environment for H steps. At each step h , the learner first decides whether to make a query. If making a query, the learner needs to select a pair of actions ( a t,h , b t,h ) ∈ A × A , upon which a feedback y t,h ∈ {-1 , 1 } is revealed to the learner regarding which action is preferred from the expert's perspective. Here the feedback is sampled according to

Irrespective of whether the learner made a query, it then picks a single action from a t,h , b t,h and transit to the next step (our algorithm will just pick an action uniformly at random from a t,h , b t,h ). After H steps, the next episode starts. Let Z t,h ∈ { 0 , 1 } indicate whether the learner decided to query at step h in episode t . We assume that the function class F is a product of H classes, i.e., F = F 0 ×···F H -1 where, for each h , we use F h = { f : X × A × A → [ -1 , 1] } to model f ⋆ h and assume that F h satisfies Assumption 1.

A policy is a mapping π : X → ∆( A ) . For a policy π , the state value function for a state x at step h is defined as V π h ( x ) := E [ ∑ H -1 i = h r i | x h = x ] and the state-action value function for a state-action pair ( x, a ) is Q π h ( x, a ) := E [ ∑ H -1 i = h r i | x h = x, a h = a ] , where the expectations are taken w.r.t. the trajectories sampled by π in the underlying MDP.

In the imitation learning setting, we assume that the expert (who gives the preference-based feedback) is equipped with a markovian policy π e , and that the preference of the expert is dependent on the

reward-to-go under π e (i.e. on a state x , actions with higher values of Q π e ( s, a ) will be preferred by the expert). Formalizing this intuition, we assume that f ⋆ h is defined such that as f ⋆ h ( x, a, b ) := Q π e h ( x, a ) -Q π e h ( x, b ) . The goal of the learner is still to minimize the regret and number of queries:

Here π t is the strategy the learner uses to select actions at episode t .

2.3 Link Function and Online Regression Oracle

Following the standard practice in the literature [Agarwal, 2013], we assume ϕ is the derivative of some α -strongly convex function (Definition 3) Φ : [ -1 , 1] → R and define the associated loss function as ℓ ϕ ( d, y ) = Φ( d ) -d ( y +1) / 2 . Additionally, in line with prior works [Foster et al., 2021, Foster and Rakhlin, 2020, Simchi-Levi and Xu, 2022, Foster et al., 2018a, Sekhari et al., 2023], our algorithm utilizes an online regression oracle, which is assumed to have a sublinear regret guarantee w.r.t. F on arbitrary data sequences.

Assumption 2. We assume the learner has access to an online regression oracle pertaining to the loss ℓ ϕ such that for any sequence { ( x 1 , a 1 , b 1 , y 1 ) , . . . , ( x T , a T , b T , y T ) } where the label y t is generated by y t ∼ ϕ ( f ⋆ ( x t , a t , b t )) , we have

for some Υ( F , T ) that grows sublinearly with respect to T . 3 For notational simplicity, whenever clear from the context, we define Υ := Υ( F , T ) .

Here Υ represents the regret upper bound and is typically of logarithmic order in T or the cardinality of the function class F in many cases (here we drop the dependence on T in notation for simplicity). We provide a few examples below.

Example 2 (Squared loss) . If we consider Φ( d ) = d 2 / 4 + d/ 2 + 1 / 4 , which is 1 / 4 -strongly convex, then we obtain ϕ ( d ) = ( d + 1) / 2 and ℓ ϕ ( d, y ) = ( d -y ) 2 / 4 , thereby recovering the squared loss, which has been widely studied in prior works. For example, Rakhlin and Sridharan [2014] characterized the minimax rates for online square loss regression in terms of the offset sequential Rademacher complexity, resulting in favorable bounds for the regret. Specifically, we have Υ = O (log |F| ) assuming the function class F is finite, and Υ = O ( d log( T )) assuming F is a d -dimensional linear class. We also kindly refer the readers to Krishnamurthy et al. [2017], Foster et al. [2018a] for efficient implementations.

Example 3 (Logistic loss) . When Φ( d ) = log(1 + exp( d )) which is strongly convex at [ -1 , 1] , we have ϕ ( d ) = 1 / (1 + exp( -d )) and ℓ ϕ ( d, y ) = log(1 + exp( -yd )) . Thus, we recover the logistic regression loss, which allows us to use online logistic regression and achieve Υ = O (log |F| ) assuming finite F . There have been numerous endeavors in minimizing the log loss, such as Foster et al. [2018b] and Cesa-Bianchi and Lugosi [2006, Chapter 9].

3 Contextual Bandits with Preference-Based Active Queries

We first present the algorithm, named AURORA, for contextual dueling bandits, as shown in Algorithm 1. At each round t ∈ [ T ] , the learner first constructs a version space F t containing all functions close to past predictors on the observed data. Here, the threshold β set to 4Υ /α +(16 + 24 α ) log ( 4 δ -1 log( T ) ) /α 2 ensures that f ⋆ ∈ F t for any t ∈ [ T ] with probability at least 1 -δ (Lemma 9). Thus, A t is non-empty for all t ∈ [ T ] and correspondingly Line 17 is well defined. The learner then forms a candidate arm set A t consisting of greedy arms induced by all functions in the version space. When |A t | = 1 , the only arm in the set is the optimal arm since f ⋆ ∈ F t , and thus no query is needed ( Z t = 0 ). However, when |A t | > 1 , any arm in A t could potentially be the optimal arm, and thus the learner needs to make a comparison query to obtain more information.

Algorithm 1 Active preference qUeRy fOR contextual bAndits (AURORA)

Require: Function class F , confidence parameter β = 4Υ α + 16+24 α α 2 log ( 4 δ -1 log( T ) ) .

1: Online regression oracle produces f 1 .

2: for t = 1 , 2 , . . . , T do

3: Learner receives context x t , computes the version space

and the candidate arm set A t ←{ π f ( x t ) : ∀ f ∈ F t } .

4: Learner decides whether to query Z t ← 1 {|A t | > 1 } .

5: if Z t = 1 then

6:

t

Set

w

sup a,b

7:

∈A

t

sup

-

t

t

Z

f,f

s

w

s

Set

λ

{

1

∑

8: if λ t = 0 then

s

1

=1

9: p t ← Uniform( A t ) .

10:

else

11:

t

γ

12:

Let

p

t

13: end if

14:

15:

.

AT/β

√

be a solution of

a

t

t

, b

∼

((

16: else

17:

Learner sets

18:

19:

20:

f

t

+1

f

t

.

end if end for

Next, we explain the learner's strategy for making queries. Firstly, the learner computes w t , which is the 'width' of the version space. Specifically, w t overestimates the instantaneous regret for playing any arm in A t (Lemma 8). Then, the learner defines λ t that indicates if the estimated cumulative regret ∑ t -1 s =1 Z s w s has exceeded √ AT/β . Note that Z t is multiplied to w t since no regret is incurred when Z t = 0 . The learner then chooses the actions (to be queried) depending on the values of λ t :

If λ t = 0 , the cumulative reward has not yet exceeded √ AT/β = O ( √ T ) , so the learner will explore as much as possible by uniform sampling from A t . √

If λ t = 1 , the regret may have reached O ( T ) , and therefore the learner uses a technique similar to inverse gap weighting (IGW), as inspired by Saha and Krishnamurthy [2022], to achieve a better balance between exploration and exploitation. Specifically, the learner solves the convex program 4 in Line 12, which is feasible and whose solution p t satisfies (Lemma 11)

As a result of the above relation, we can convert the instantaneous regret to the point-wise error between the predictor f t and the truth f ⋆ plus an additive A/γ t . This allows us to bound the cumulative point-wise error by the regret of the online regression oracle. In the special case where there exists a 'reward function' r : X × A → [0 , 1] for each f ∈ F such that f ( x, a, b ) = r ( x, a ) -r ( x, b ) (Example 1), the solution p t can be directly written as

̸

̸

where r t is the reward function associated with f t , i.e., f t ( x, a, b ) = r t ( x, a ) -r t ( x, b ) . This is the standard IGW exploration strategy [Foster and Rakhlin, 2020] and leads to the same guarantee as (1) (see Lemma 12).

x

, a

t

t

and

t

b

′

∈F

t

t

f

(

x

, a, b

√

AT/β

)

}

.

∑

t

t

, a, b

)

p

(

b

) +

p

t

a

∈A

b

f

t

(

-

x

t

γ

t

2

p

t

(

a

)

independently and receives the feedback

, b

t

t

)

t

)

, y

5

A

γ

t

y

t

.

.

to the online regression oracle which returns to be the only action in

A

t

, and plays them.

′

a

f

(

x

t

, a, b

)

f

t

+1

.

Wediscuss the computational tractability of Algorithm 1 in Appendix A. In short, it is computationally tractable for some structured function classes (e.g. linear and tabular function classes). For general function classes, it is also efficient given a regression oracle.

3.1 Theoretical Analysis

Towards the statistical guarantees of Algorithm 1, we employ two quantities to characterize a contextual bandit instance: the uniform gap and the eluder dimension, which are introduced below.

̸

Assumption 3 (Uniform gap) . We assume the optimal arm π f ⋆ ( x ) induced by f ⋆ under any context x ∈ X is unique. Further, we assume a uniform gap ∆ := inf x inf a = π f ⋆ ( x ) f ⋆ ( x, π f ⋆ ( x ) , a ) > 0 .

We note that the existence of a uniform gap is a standard assumption in the literature of contextual bandits [Dani et al., 2008, Abbasi-Yadkori et al., 2011, Audibert et al., 2010, Garivier et al., 2019, Foster and Rakhlin, 2020, Foster et al., 2021]. Next, we introduce the eluder dimension [Russo and Van Roy, 2013] and begin by defining ' ϵ -dependence'.

Definition 1 ( ϵ -dependence) . Let G ⊆ X → R be any function class. We say an element x ∈ X is ϵ -dependent on { x 1 , x 2 , . . . , x n } ⊆ X with respect to G if any pair of functions g, g ′ ∈ G satisfying ∑ n i =1 ( g ( x i ) -g ′ ( x i )) 2 ≤ ϵ 2 also satisfies g ( x ) -g ′ ( x ) ≤ ϵ . Otherwise, we say x is ϵ -independent of { x 1 , x 2 , . . . , x n } .

Definition 2 (Eluder dimension) . The ϵ -eluder dimension of a function class G ⊆ X → R , denoted by dim E ( G , ϵ ) , is the length d of the longest sequence of elements in X satisfying that there exists some ϵ ′ ≥ ϵ such that every element in the sequence is ϵ ′ -independent of its predecessors.

Eluder dimension is a complexity measure for function classes and has been used in the literature of bandits and RL extensively [Chen et al., 2022, Osband and Van Roy, 2014, Wang et al., 2020, Foster et al., 2021, Wen and Van Roy, 2013, Jain et al., 2015, Ayoub et al., 2020, Ishfaq et al., 2021, Huang et al., 2022]. Examples where the eluder dimension is small include linear functions, generalized linear models, and functions in Reproducing Kernel Hilbert Space (RKHS).

Given these quantities, we are ready to state our main results. The proofs are provided in Appendix C. Theorem 1. Under Assumptions 1 to 3, Algorithm 1 guarantees the following upper bounds of the regret and the number of queries:

with probability at least 1 -δ . We recall that β = O ( α -1 Υ+ α -2 log( δ -1 log( T ))) , and α represents the coefficient of strong convexity of Φ . Logarithmic terms are hidden in the upper bounds for brevity.

For commonly used loss function (Examples 2 and 3), β only scales logarithmically with the size (or effective size) of F and is thus mild. For instance, for a finite class F , β depends on log |F| . Furthermore, when F is infinite, we can replace |F| by the covering number of F following the standard techniques: for d -dimensional linear function class, β will have a dependence of O ( d ) (effective complexity of F ), and for the tabular class, β will have a dependence of O ( SA 2 ) . In other words, the rate of β is acceptable as long as the complexity of the function class F is acceptable.

Best-of-both-worlds guarantee. We observe that both the regret bound and the query complexity bound consist of two components: the worst-case bound and the instance-dependent bound. The worst-case bound provides a guarantee under all circumstances, while the instance-dependent one may significantly improve the upper bound when the underlying problem is well-behaved (i.e., has a small eluder dimension and a large gap).

Lower bounds. To see whether these upper bounds are tight, we provide a lower bound which follows from a reduction from regular multi-armed bandits to contextual dueling bandits.

Theorem 2 (Lower bounds) . The following two claims hold: (1) For any algorithm, there exists an instance that leads to Regret CB T = Ω( √ AT ) ; (2) For any algorithm achieving a worse-case expected regret upper bound in the form of E [Regret CB T ] = O ( √ AT ) , there exists an instance with gap ∆ = √ A/T that results in E [Regret CB T ] = Ω( A/ ∆) and E [Queries CB T ] = Ω( A/ ∆ 2 ) = Ω( T ) .

By relating these lower bounds to Theorem 1, we conclude that our algorithm achieves a tight dependence on the gap ∆ and the number of rounds T up to logarithmic factors. Furthermore, as an additional contribution, we establish an alternative lower bound in Section C.4.1 by conditioning on the limit of regret, rather than the worst-case regret as assumed in Theorem 2.

Intuition of proofs. We next provide intuition for why our algorithm has the aforementioned theoretical guarantees. First, we observe that from the definition of λ t , the left term inside the indicator is non-decreasing, which allows us to divide rounds into two phases. In the first phase, λ t is always 0, and then at some point, it changes to 1 and remains 1 for the rest rounds. After realizing this, we first explain the intuition of the worst-case regret. In the first phase, as w t is an overestimate of the instantaneous regret (see Lemma 8), the accumulated regret in this phase cannot exceed O ( √ T ) . In the second phase, we adapt the analysis of IGW to this scenario to obtain an O ( √ T ) upper bound. A similar technique has been used in Saha and Krishnamurthy [2022], Foster et al. [2021]. As the regret in both phases is at most O ( √ T ) , the total regret cannot exceed O ( √ T ) . Next, we explain the intuition of instance-dependent regret. Due to the existence of a uniform gap ∆ , we can first prove that as long as |A t | > 1 , we must have w t ≥ ∆ (see Lemma 7). This means that for all rounds that may incur regret, the corresponding width is at least ∆ . However, this cannot happen too many times as this frequency is bounded by the eluder dimension, which leads to an instance-dependent regret upper bound. Leveraging a similar technique, we can also obtain an upper bound on the number of queries.

Comparison to regret bounds of dueling bandits. As established by prior works [Yue et al., 2012, Saha and Gaillard, 2022], for dueling bandits, the minimax regret rate is ˜ Θ( √ AT ) and the instance-dependent rate is ˜ Θ( A/ ∆) . If we reduce our result (Theorem 1) into the dueling bandits setting, we will get

where the second equality holds since the eluder dimension is upper bounded by A for dueling bandits. We observe that the worst-case regret rate is the same, but there is a gap of A 2 in the instance-dependent bound. The improvement of this gap is an interesting future direction.

Comparion to MINMAXDB [Saha and Krishnamurthy, 2022]. In this prior work, the authors assume that Pr( y = 1 | x, a, b ) = ( f ⋆ ( x, a, b ) + 1) / 2 , which is a specification of our feedback model (Example 2). While our worst-case regret bound matches their regret bound, our paper improves upon their results by having an additional instance-dependent regret bound that depends on the eluder dimension and gap. Furthermore, we also provide bounds on the query complexity which could be small for benign instances while MINMAXDB simply queries on every round.

Comparion to ADACB [Foster et al., 2021]. Our method shares some similarities with Foster et al. [2021], especially in terms of theoretical results, but differs in two aspects: (1) they assume regular contextual bandits where the learner observes the reward directly, while we assume preference feedback, and (2) they assume a stochastic setting where contexts are drawn i.i.d., but we assume that the context is adversarially chosen. While these two settings may not be directly comparable, it should be noted that [Foster et al., 2021] do not aim to minimize query complexity.

Results without the uniform gap assumption. We highlight that Theorem 1 can naturally extend to scenarios where a uniform gap does not exist (i.e., when Assumption 3 is not satisfied) without any modifications to the algorithm. The result is stated below, which is analogous to Theorem 1.

Theorem 3. Under Assumptions 1 and 2, Algorithm 1 guarantees the following upper bounds of the regret and the number of queries:

̸

with probability at least 1 -δ . Here we define the gap of context x as Gap( x ) := min a = π f ⋆ ( x ) f ⋆ ( x, π f ⋆ ( x ) , a ) and the number of rounds where contexts have small gap as T ϵ := ∑ T t =1 1 { Gap( x t ) ≤ ϵ } . We also recall that β = O ( α -1 Υ+ α -2 log( δ -1 log( T ))) , and α denotes the coefficient of strong convexity of Φ .

Compared to Theorem 1, the above result has an extra gap-dependent term, T ϵ , measuring how many times the context falls into a small-gap region. We highlight that T ϵ is small under certain conditions such as the Tsybakov noise condition [Tsybakov, 2004]. It is also worth mentioning that our algorithm is agnostic to ϵ , thus allowing us to take the minimum over all ϵ > 0 .

Comparion to SAGE-BANDIT [Sekhari et al., 2023]. Theorem 3 is similar to Theorem 4 in Sekhari et al. [2023], which studies active queries in contextual bandits with standard reward signal. Although our result looks slightly worse in terms of the factor A , we believe that this inferiority is reasonable since our approach requires two actions to form a query, thus analytically expanding the action space to A 2 . Whether this dependency can be improved remains a question for future investigation.

4 Imitation Learning with Preference-Based Active Queries

In this section, we introduce our second algorithm, which is presented in Algorithm 2 for imitation learning. In essence, the learner treats the MDP as a concatenation of H contextual bandits and runs an instance of AURORA (Algorithm 1) for each time step. Specifically, the learner first creates H instances of AURORA, denoted by AURORA h (for h = 0 , . . . , H -1 ). Here, AURORA h should be thought of as an interactive program that takes the context x as input and outputs a , b , and Z . At each episode t , and each step h therein, the learner first feeds the current state x t,h to AURORA h as the context; then, AURORA h decides whether to query (i.e. Z t,h ) and returns the actions a t,h and b t,h . If it decides to make a query, the learner will ask for the feedback y t,h on the proposed actions a t,h , b t,h , and provide the information (( x t,h , a t,h , b t,h ) , y t,h ) back to AURORA h to update its online regression oracle (and other local variables). We recall that the noisy binary feedback y t,h is sampled as y t,h ∼ ϕ ( Q π e h ( x t,h , a t,h ) -Q π e h ( x t,h , b t,h )) , and also emphasize that the learner neither has access to a ∼ π e ( x t,h ) like in DAGGER [Ross et al., 2011] nor reward-to-go like in AGGREVATE(D) [Ross and Bagnell, 2014, Sun et al., 2017]. Finally, the learner chooses one of the two actions uniformly at random, executes it in the underlying MDP, and transits to the next state x t,h +1 in the episode. The above process is then repeated with AURORA h +1 till the episode ends. We name this algorithm AURORAE, the plural form of AURORA, which signifies that the algorithm is essentially a stack of multiple AURORA instances.

4.1 Theoretical Analysis

As Algorithm 2 is essentially a stack of Algorithm 1, we can inherit many of the theoretical guarantees from the previous section. To state the results, we first extend Assumption 3 into imitation learning.

̸

Assumption 4 (Uniform Gap) . Let f ⋆ h be defined such that for any x ∈ X , a, b ∈ A 2 , f ⋆ h ( x, a, b ) = Q π e h ( x, a ) -Q π e h ( x, b ) . For all h , we assume the optimal action for f ⋆ h under any state x ∈ X is unique. Further, we assume a uniform gap ∆ := inf h inf x inf a = π f ⋆ h ( x ) f ⋆ h ( x, π f ⋆ h ( x ) , a ) > 0 .

This assumption essentially says that Q π e h has a gap in actions. We remark that, just as Assumption 3 is a common condition in the bandit literature, Assumption 4 is also common in MDPs [Du et al.,

2019, Foster et al., 2021, Simchowitz and Jamieson, 2019, Jin and Luo, 2020, Lykouris et al., 2021, He et al., 2021]. The theoretical guarantee for Algorithm 2 is presented in Theorem 4. We note a technical difference between this result and Theorem 1: although we treat the MDP as a concatenation of H contextual bandits, the instantaneous regret of imitation learning is defined as the performance gap between the combined policy π t derived from the H instances as a cohesive unit and the expert policy. This necessitates the use of performance difference lemma (Lemma 5) to get a unified result.

Theorem 4. Under Assumptions 1, 2 and 4, Algorithm 2 guarantees the following upper bounds of the regret and the number of queries:

with probability at least 1 -δ . Here Adv T := ∑ T t =1 ∑ H -1 h =0 E x t,h ∼ d πt x t, 0 ,h [max a A π e h ( x t,h , a )] is non-negative, and d π t x t, 0 ,h ( x ) denotes the probability of π t 5 reaching the state x at time step h starting from inital state x t, 0 . In the above, β = O ( α -1 Υ + α -2 log( Hδ -1 log( T ))) and α denotes the coefficient of strong convexity of Φ .

Compared to Theorem 1, the main terms of the upper bounds for imitation learning are precisely the bounds in Theorem 1 multiplied by H . In the proof presented in Appendix C.6, we use the performance difference lemma to reduce the regret of imitation learning to the sum of the regret of H contextual dueling bandits, which explains this additional factor of H .

Another interesting point is that the main term of the regret upper bound is subtracted by a non-negative term Adv T , which measures the degree to which we can outperform the expert policy. In other words, our algorithm not only competes with the expert policy but can also surpass it to some extent. To see this, let us consider the average regret , which is defined as AveRegret IL T := Regret IL T /T = ∑ T t =1 ( V π e 0 ( x t, 0 ) -V π t 0 ( x t, 0 )) /T. Then, Theorem 4 implies that AveRegret IL T = O ( H √ Aβ/T ) -Adv T /T where we have simplified it by ignoring the instance-dependent upper bound and logarithmic factors for clarity. Now, consider a case where max a A π e h ( x, a ) > α 0 for some constant α 0 > 0 for all x and h . This can happen when the expert policy is suboptimal for every state. Consequently, we have Adv T > α 0 HT . In this case, the average regret is further bounded by AveRegret IL T = O ( H √ Aβ/T ) -α 0 H. When T → ∞ , we have AveRegret IL T →-α 0 H < 0 . This means that the best (or average) learned policy will eventually outperform the expert policy. This guarantee is stronger than that of DAGGER [Ross et al., 2011] in that DAGGER cannot ensure the learned policy is better than the expert policy regardless of how suboptimal the expert may be. While this may look surprising at first glance since we are operating under a somewhat weaker query mode than that of DAGGER, we note that by querying experts for comparisons on pairs of actions with feedback sampling as y ∼ ϕ ( Q π e ( x, a ) -Q π e ( x, b )) , it is possible to identify the action that maximizes Q π e ( x, a ) (even if we cannot identify the value Q π e ( x, a ) ). Finally, we remark that our worst-case regret bound is similar to that of Ross and Bagnell [2014], Sun et al. [2017], which can also outperform a suboptimal expert but require access to both expert's actions and reward signals - a much stronger query model than ours.

5 Conclusion

We presented interactive decision-making algorithms that learn from preference-based feedback while minimizing query complexity. Our algorithms for contextual bandits and imitation learning share worst-case regret bounds similar to the bounds of the state-of-the-art algorithms in standard settings while maintaining instance-dependent regret bounds and query complexity bounds. Notably, our imitation learning algorithm can outperform suboptimal experts, matching the result of Ross and Bagnell [2014], Sun et al. [2017], which operates under much stronger feedback.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.

Alekh Agarwal. Selective sampling algorithms for cost-sensitive multiclass prediction. In International Conference on Machine Learning , pages 1220-1228. PMLR, 2013.

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , pages 10-4, 2019.

Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing dueling bandits to cardinal bandits. In International Conference on Machine Learning , pages 856-864. PMLR, 2014.

Jean-Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In COLT , pages 41-53, 2010.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR, 2020.

Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hüllermeier. Preference-based online learning with dueling bandits: A survey. The Journal of Machine Learning Research , 22(1): 278-385, 2021.

Viktor Bengs, Aadirupa Saha, and Eyke Hüllermeier. Stochastic contextual dueling bandits under linear stochastic transitivity models. In International Conference on Machine Learning , pages 1764-1786. PMLR, 2022.

Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Conference on robot learning , pages 519-528. PMLR, 2018.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning , pages 783-792. PMLR, 2019.

Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning , pages 330-359. PMLR, 2020.

Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preferencebased reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm. Machine learning , 97:327-351, 2014.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Nicolo Cesa-Bianchi, Gábor Lugosi, and Gilles Stoltz. Minimizing regret with label efficient prediction. IEEE Transactions on Information Theory , 51(6):2152-2162, 2005.

Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. Learning to search better than your teacher. In International Conference on Machine Learning , pages 20582066. PMLR, 2015.

Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning , pages 3773-3793. PMLR, 2022.

Ching-An Cheng and Byron Boots. Convergence of value aggregation for imitation learning. In International Conference on Artificial Intelligence and Statistics , pages 1801-1809. PMLR, 2018.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceedings of the 22nd international conference on Machine learning , pages 137-144, 2005.

Robert Cohn, Edmund Durfee, and Satinder Singh. Comparing action-query strategies in semiautonomous agents. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 25, pages 1102-1107, 2011.

Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. In 21st Annual Conference on Learning Theory , pages 355-366, 2008.

Hal Daumé, John Langford, and Daniel Marcu. Search-based structured prediction. Machine learning , 75:297-325, 2009.

Ofer Dekel, Claudio Gentile, and Karthik Sridharan. Selective sampling and active learning from single and multiple experts. The Journal of Machine Learning Research , 13(1):2655-2697, 2012.

Simon S Du, Yuping Luo, Ruosong Wang, and Hanrui Zhang. Provably efficient q-learning with function approximation via distribution shift error checking oracle. Advances in Neural Information Processing Systems , 32, 2019.

Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory , pages 563-587. PMLR, 2015.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning , pages 3199-3210. PMLR, 2020.

Dylan Foster, Alekh Agarwal, Miroslav Dudík, Haipeng Luo, and Robert Schapire. Practical contextual bandits with regression oracles. In International Conference on Machine Learning , pages 1539-1548. PMLR, 2018a.

Dylan Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. In Conference on Learning Theory , pages 2059-2059. PMLR, 2021.

Dylan J Foster, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Logistic regression: The importance of being improper. In Conference On Learning Theory , pages 167-208. PMLR, 2018b.

Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm. Machine learning , 89:123-156, 2012.

Aurélien Garivier, Pierre Ménard, and Gilles Stoltz. Explore first, exploit next: The true shape of regret in bandit problems. Mathematics of Operations Research , 44(2):377-399, 2019.

Steve Hanneke and Liu Yang. Minimax analysis of active learning. J. Mach. Learn. Res. , 16(1): 3487-3602, 2015.

Steve Hanneke and Liu Yang. Toward a general theory of online selective sampling: Trading off mistakes and queries. In International Conference on Artificial Intelligence and Statistics , pages 3997-4005. PMLR, 2021.

Jiafan He, Dongruo Zhou, and Quanquan Gu. Logarithmic regret for reinforcement learning with linear function approximation. In International Conference on Machine Learning , pages 41714180. PMLR, 2021.

Baihe Huang, Jason D Lee, Zhaoran Wang, and Zhuoran Yang. Towards general function approximation in zero-sum markov games. In International Conference on Learning Representations , 2022.

Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang, Zhaoran Wang, Doina Precup, and Lin Yang. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning , pages 4607-4616. PMLR, 2021.

Ashesh Jain, Shikhar Sharma, Thorsten Joachims, and Ashutosh Saxena. Learning preferences for manipulation tasks from online coactive feedback. The International Journal of Robotics Research , 34(10):1296-1313, 2015.

Tiancheng Jin and Haipeng Luo. Simultaneously learning stochastic and adversarial episodic mdps with known transition. Advances in neural information processing systems , 33:16557-16566, 2020.

Sham M Kakade and Ambuj Tewari. On the generalization ability of online strongly convex programming algorithms. Advances in Neural Information Processing Systems , 21, 2008.

Junpei Komiyama, Junya Honda, Hisashi Kashima, and Hiroshi Nakagawa. Regret lower bound and optimal algorithm in dueling bandit problem. In Conference on learning theory , pages 1141-1154. PMLR, 2015.

Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. In International Conference on Machine Learning , pages 1915-1924. PMLR, 2017.

Michael Laskey, Sam Staszak, Wesley Yu-Shu Hsieh, Jeffrey Mahler, Florian T Pokorny, Anca D Dragan, and Ken Goldberg. Shiv: Reducing supervisor burden in dagger using support vectors for efficient learning from demonstrations in high dimensional state spaces. In 2016 IEEE International Conference on Robotics and Automation (ICRA) , pages 462-469. IEEE, 2016.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021a.

Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning , pages 6152-6163. PMLR, 2021b.

Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv preprint arXiv:2305.18438 , 2023.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 , 2023.

Thodoris Lykouris, Max Simchowitz, Alex Slivkins, and Wen Sun. Corruption-robust exploration in episodic reinforcement learning. In Conference on Learning Theory , pages 3242-3245. PMLR, 2021.

Vivek Myers, Erdem Bıyık, and Dorsa Sadigh. Active reward learning from online preferences. arXiv preprint arXiv:2302.13507 , 2023.

Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for preference-based reinforcement learning. In Conference on Uncertainty in Artificial Intelligence , pages 1029-1038. PMLR, 2020.

OpenAI. Gpt-4 technical report. arXiv , 2023.

Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends® in Robotics , 7(1-2): 1-179, 2018.

Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems , 27, 2014.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35: 27730-27744, 2022.

Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: reinforcement learning with trajectory preferences. arXiv preprint arXiv:2111.04850 , 2021.

Alexander Rakhlin and Karthik Sridharan. Online non-parametric regression. In Conference on Learning Theory , pages 1232-1264. PMLR, 2014.

Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979 , 2014.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

Stéphane Ross, Narek Melik-Barkhudarov, Kumar Shaurya Shankar, Andreas Wendel, Debadeepta Dey, J Andrew Bagnell, and Martial Hebert. Learning monocular reactive uav control in cluttered natural environments. In 2013 IEEE international conference on robotics and automation , pages 1765-1772. IEEE, 2013.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions . 2017.

Aadirupa Saha. Optimal algorithms for stochastic contextual preference bandits. Advances in Neural Information Processing Systems , 34:30050-30062, 2021.

Aadirupa Saha and Pierre Gaillard. Dueling bandits with adversarial sleeping. Advances in Neural Information Processing Systems , 34:27761-27771, 2021.

Aadirupa Saha and Pierre Gaillard. Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences. In International Conference on Machine Learning , pages 19011-19026. PMLR, 2022.

Aadirupa Saha and Akshay Krishnamurthy. Efficient and optimal algorithms for contextual dueling bandits under realizability. In International Conference on Algorithmic Learning Theory , pages 968-994. PMLR, 2022.

Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Dueling rl: Reinforcement learning with trajectory preferences. In International Conference on Artificial Intelligence and Statistics , pages 6263-6289. PMLR, 2023.

Ayush Sekhari, Karthik Sridharan, Wen Sun, and Runzhe Wu. Selective sampling and imitation learning via online regression. arXiv preprint arXiv:2307.04998 , 2023.

David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research , 47(3):1904-1931, 2022.

Max Simchowitz and Kevin G Jamieson. Non-asymptotic gap-dependent regret bounds for tabular mdps. Advances in Neural Information Processing Systems , 32, 2019.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International conference on machine learning , pages 3309-3318. PMLR, 2017.

Aleksandar Taranovic, Andras Gabor Kupcsik, Niklas Freymuth, and Gerhard Neumann. Adversarial imitation learning with preferences. In The Eleventh International Conference on Learning Representations , 2022.

Alexander B Tsybakov. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics , 32(1):135-166, 2004.

Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135, 2020.

Zheng Wen and Benjamin Van Roy. Efficient exploration and value function generalization in deterministic systems. Advances in Neural Information Processing Systems , 26, 2013.

Christian Wirth and Johannes Fürnkranz. On learning from game annotations. IEEE Transactions on Computational Intelligence and AI in Games , 7(3):304-316, 2014.

Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preferencebased reinforcement learning methods. Journal of Machine Learning Research , 18(136):1-46, 2017.

Huasen Wu and Xin Liu. Double thompson sampling for dueling bandits. Advances in neural information processing systems , 29, 2016.

Yue Wu, Tao Jin, Hao Lou, Farzad Farnoud, and Quanquan Gu. Borda regret minimization for generalized linear dueling bandits. arXiv preprint arXiv:2303.08816 , 2023.

Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems , 33:18784-18794, 2020.

Yisong Yue and Thorsten Joachims. Beat the mean bandit. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 241-248. Citeseer, 2011.

Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences , 78(5):1538-1556, 2012.

Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. How to query human feedback efficiently in rl? arXiv preprint arXiv:2305.18505 , 2023.

David Zhang, Micah Carroll, Andreea Bobu, and Anca Dragan. Time-efficient reward learning via visually assisted cluster ranking. arXiv preprint arXiv:2212.00169 , 2022.

Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or k -wise comparisons. In International Conference on Machine Learning . PMLR, 2023.

Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. Advances in Neural Information Processing Systems , 35:35379-35391, 2022.

Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd national conference on Artificial intelligenceVolume 3 , pages 1433-1438, 2008.

Masrour Zoghi, Shimon Whiteson, Remi Munos, and Maarten Rijke. Relative upper confidence bound for the k-armed dueling bandit problem. In International conference on machine learning , pages 10-18. PMLR, 2014.

A Computational tractability of Algorithm 1

We observe that the computational complexity of the proposed algorithm mainly depends on the computation of the candidate arm set (Line 3) and the width (Line 6). When F is a d -dimensional linear function class, the computational complexity can be ˜ O ( dTA ) since the version space exhibits an ellipsoid structure and thus both the candidate arm and the width can be computed in ˜ O ( dA ) time. When F is tabular, it can be considered as a special case of linear class with one-hot encoding. In this case, we have d = S × A , resulting in a computational complexity of ˜ O ( TSA 2 ) .

For a more general convex function class F , we can design an efficient algorithm based on a weighted regression oracle for F . To this end, we first note that an approach to efficiently compute the width has been proposed by Foster et al. [2018a], and now we propose the following method to compute the candidate arm set. An arm a belongs to the candidate arm set at round t if and only if the following optimization problem (with an constant objective) is feasible

̸

Here we introduce the slack variable ξ so that the optimization part for f can be simply reduced to a weighted regression oracle. Next, we convert the above into Lagrangian formulation and obtain

̸

̸

̸

̸

Here we can swap the min and max since the objective is convex in the joint space of f and ξ . Then, the inner minimization problem can be solved by updating f via the regression oracle and updating ξ via gradient descent; for the outer maximization problem, we can do projected gradient ascent.

B Preliminaries

Lemma 2 (Kakade and Tewari [2008, Lemma 3]) . Suppose X 1 , . . . , X T is a martingale difference sequence with | X t | ≤ b . Let

Let V = ∑ T t =1 Var t X t be the sum of conditional variances of X t 's. Further, let σ = √ V . Then we have, for any δ < 1 /e and T ≥ 3 ,

Lemma 3 (Foster and Rakhlin [2020, Lemma 3]) . For any vector ˆ y ∈ [0 , 1] A , if we define p to be

̸

̸

where ˆ a = arg max a ˆ y ( a ) , then for any y ⋆ ∈ [0 , 1] A and γ > 0 , we have

Lemma 4 (Zhu and Nowak [2022, Lemma 2]) . Let ( Z t ) t ≤ T to be real-valued sequence of positive random variables adapted to a filtration F t . If | Z t | ≤ B almost surely, then with probability at least 1 -δ ,

Lemma 5 (Performance difference lemma [Agarwal et al., 2019]) . For any two policies π and π ′ and any state x 0 ∈ X , we have

where A π h ( x, a ) = Q π h ( x, a ) -V π h ( x, a ) and d π x 0 ,h ( x, a ) is the probability of π reaching the stateaction pair ( x, a ) at time step h starting from initial state x 0 .

Lemma 6. For any two Bernoulli distributions Bern( x ) and Bern( y ) with x, y ∈ [ b, 1 -b ] for some 0 < b ≤ 1 / 2 , the KL divergence is bounded as

Proof of Lemma 6. Denote ∆ = x -y . Then, by definition, we have

Since ln(1 + x ) ≤ x for all x > -1 , we have

C Missing Proofs

C.1 Supporting Lemmas

Definition 3 (Strong convexity) . A function Φ : [ -1 , 1] → R is α -strongly-convex if for all u, u ′ ∈ R , we have

where ∇ Φ means the derivative of Φ .

Lemma 7. For any t ∈ [ T ] , if f ⋆ ∈ F t , then we have w t ≥ ∆ whenever |A t | > 1 .

and

Proof of lemma 7. When |A t | > 1 , we know there exists a function f ′ ∈ F t satisfying

̸

Then we have ∆ ≤ f ⋆ ( x t , a ⋆ t , a ′ ) ≤ f ⋆ ( x t , a ⋆ t , a ′ ) -f ′ ( x t , a ⋆ t , a ′ ) ≤ w t where the second inequality holds since f ′ ( x t , a ⋆ t , a ′ ) ≤ 0 .

Lemma 8. For any t ∈ [ T ] and any arm a ∈ A t , we have f ⋆ ( x t , π f ⋆ ( x t ) , a ) ≤ w t .

Proof of Lemma 8. For any a ∈ A t , by the definition of A t , there must exists a function f for which a = π f ( x t ) . Hence,

The following lemma is adapted from Agarwal [2013, Lemma 2].

Lemma 9. The following holds with probability at least 1 -δ for any T > 3 ,

Proof of Lemma 9. Throughout the proof, we denote z t := ( x t , a t , b t ) for notational simplicity. We define D Φ as the Bregman divergence of the function Φ :

where we recall that ϕ = Φ ′ is the derivative of Φ . Since Φ is α -strong convex, we have α ( u -v ) 2 / 2 ≤ D Φ ( u, v ) , and hence,

Hence, it suffice to derive an upper bound for the Bregman divergence in the right hand side above. Define ν t as below:

We note that E t [( y t + 1) / 2] = ϕ ( f ⋆ ( z t )) , and thus E t [ ν t ] = 0 , which means ν t is a martingale difference sequence. Now we bound the value and the conditional variance of ν t in order to derive concentration results.

Bound the value of ν t :

Bound the conditional variance of ν t :

where for the last line we note that x t , g t are measurable at t .

Now we apply Lemma 2, which yields for any δ < 1 /e and T > 3 , with probability at least 1 -4 δ log( T ) ,

Recall the definition of ν t , and we conclude that

which implies

Plugging this upper bound of Bregman divergence into (2), we obtain that, with probability at least 1 -4 δ log( T ) , for any δ < 1 /e and T > 3 , we have

Finally, we finish the proof by adjusting the coefficient δ and taking a union bound to obtain the desired result.

The following lemma is a variant of Russo and Van Roy [2013, Proposition 3], with the main difference being that (1) the version space is established using the function produced by the oracle instead of the least squares estimator, and (2) the extra multiplicative factor Z t .

Lemma 10. For Algorithm 1, it holds that

for any constant ϵ > 0 ,

Proof of Lemma 10. We first define a subsequence consisting only of the elements for which we made a query in that round. Specifically, we define (( x i 1 , a i 1 , b i 1 ) , ( x i 2 , a i 2 , b i 2 ) , . . . , ( x i k , a i k , b i k )) where 1 ≤ i 1 < i 2 < · · · < i k ≤ T and ( x t , a t , b t ) belongs to the subsequence if and only if Z t = 1 . We further simplify the notation by defining z j := ( x i j , a i j , b i j ) and f ( z j ) := f ( x i j , a i j , b i j ) . Then we note that the left-hand side of (3) is equivalent to

and the version space in Algorithm 1 is equal to

Hence, it suffice to establish the lower bound for (4) under the version space of (5). To that end, we make one more simplification in notation: we denote

We begin by showing that if w ′ j > ϵ for some j ∈ [ k ] , then z j is ϵ -dependent on at most 4 β/ϵ 2 disjoint subsequence of its predecessors. To see this, we note that when w ′ j > ϵ , there must exist two function f, f ′ ∈ F j such that f ( z j ) -f ′ ( z j ) > ϵ . If z j is ϵ -dependent on a subsequence ( z i 1 , z i 2 , . . . , z i n ) of its predecessors, we must have

Hence, if z j is ϵ -dependent on l disjoint subsequences, we have

For the left-hand side above, we also have

where the first inequality holds since ( a + b ) 2 ≤ 2( a 2 + b 2 ) for any a, b , and the second inequality holds by (5). Combining (6) and (7), we get that l ≤ 4 β/ϵ 2 .

Next, we show that for any sequence ( z ′ 1 , . . . , z ′ τ ) , there is at least one element that is ϵ -dependent on at least τ/d -1 disjoint subsequence of its predecessors, where d := dim E ( F , ϵ ) . To show this, let m be the integer satisfying md +1 ≤ τ ≤ md + d . Wewill construct m disjoint subsequences, B 1 , . . . , B m . At the beginning, let B i = ( z ′ i ) for i ∈ [ m ] . If z ′ m +1 is ϵ -dependent on each subsequence B 1 , . . . , B m , then we are done. Otherwise, we select a subsequence B i which z ′ m +1 is ϵ -independent of and append z ′ m +1 to B i . We repeat this process for all elements with indices j > m + 1 until either z ′ j is ϵ -dependent on each subsequence or j = τ . For the latter, we have ∑ m i =1 | B i | ≥ md , and since each element of a subsequence B i is ϵ -independent of its predecesors, we must have | B i | = d for all i . Then, z τ must be ϵ -dependent on each subsequence by the definition of eluder dimension.

Finally, let's take the sequence ( z ′ 1 , . . . z ′ τ ) to be the subsequence of ( z 1 , . . . , z k ) consisting of elements z j for which w ′ j > ϵ . As we have established, we have (1) each z ′ j is ϵ -dependent on at most 4 β/ϵ 2 disjoint subsequences, and (2) some z ′ j is ϵ -dependent on at least τ/d -1 disjoint subsequences. Therefore, we must have τ/d -1 ≤ 4 β/ϵ 2 , implying that τ ≤ (4 β/ϵ 2 +1) d .

The following lemma is adopted from Saha and Krishnamurthy [2022, Lemma 3].

Lemma 11. For any function f ∈ F and any context x ∈ X , the following convex program of p ∈ ∆( A ) is always feasible:

Furthermore, any solution p satisfies:

whenever γ ≥ 2 A .

Lemma 12. Assume that for each f ∈ F , there exists an associated function r : X × A → [0 , 1] such that f ( x, a, b ) = r ( x, a ) -r ( x, b ) for any x ∈ X and a, b ∈ A . In this case, for any context x ∈ X , if we define p as

̸

̸

then we have

Proof of lemma 12. Fix any b ∈ A . Then, the distribution p can be rewritten as

̸

̸

Therefore, denoting f ⋆ ( x, a, b ) = r ⋆ ( x, a ) -r ⋆ ( x, b ) for some function r ⋆ , we have

where for the inequality above we invoked Lemma 3 with ˆ y ( a ) = ( r ( x, a ) -r ( x, b ) + 1) / 2 and y ⋆ ( a ) = ( r ⋆ ( x, a ) -r ⋆ ( x, b ) + 1) / 2 . We note that the above holds for any b ∈ A . Hence, we complete the proof by sampling b ∼ p .

Lemma 13. Assume f ⋆ ∈ F t for all t ∈ [ T ] . Suppose there exists some t ′ ∈ [ T ] such that λ t = 0 for all t ≤ t ′ . Then we have

with probability at least 1 -δ .

Proof. Since f ⋆ ∈ F t , we always have π f ⋆ ( x t ) ∈ A t for all t ∈ [ T ] . Hence, whenever Z t is zero, we have A t = { π f ⋆ ( x t ) } and thus we do not incur any regret. Hence, we know Z t w t is either 0 or at least ∆ by Lemma 7. Let us fix an integer m> 1 / ∆ , whose value will be specified later. We divide the interval [∆ , 1] into bins of width 1 /m and conduct a refined study of the sum of Z t w t :

where in the third inequality we replace the supremum over a, b by the summation over a, b , and in the last inequality we further replace it by the expectation. Here recall that p t ( a ) is uniform when λ t = 0 , leading to the extra A 2 factor. To deal with ( ∗ ) , we first apply Lemma 4 to recover the empirical a t and b t , and then apply Lemma 10 to get an upper bound via the eluder dimension:

with probability at least 1 -δ . Plugging ( ∗ ) back, we obtain

where for the second inequality, we use the fact that ( j +1) /m ≤ 2 j/m for any j ≥ 1 ; for the third inequality, we assume m> 1 / ∆ . Setting m = 2 / ∆ , we arrive at

which completes the proof.

Lemma 14. Whenever

we have λ 1 = λ 2 = · · · = λ T = 0 with probability at least 1 -δ .

Proof of Lemma 14. We prove it via contradiction. Assume the inequality holds but there exists t ′ for which λ t ′ = 1 . Without loss of generality, we assume that λ t = 0 for all t < t ′ , namely that t ′ is the first time that λ t is 1. Then by definition of λ t ′ , we have

On the other hand, by Lemma 13, we have

The combination of the above two inequalities contradicts with the conditions.

C.2 Proof of Lemma 1

Proof of Lemma 1. We prove it via contradiction. If no such arm exists, meaning that for any arm a , there exists an arm b such that f ⋆ ( x, a, b ) < 0 . Then we can find a sequence of arms ( a 1 , a 2 , . . . , a k ) such that f ⋆ ( x, a i , a i +1 ) < 0 for any i = 1 , . . . , k -1 and f ⋆ ( x, a k , a 1 ) < 0 , which contradicts with the transitivity (Assumption 1).

C.3 Proof of Theorem 1

We begin by showing the worst-case regret upper bound.

Lemma 15 (Worst-case regret upper bound) . For Algorithm 1, assume f ⋆ ∈ F t for all t ∈ [ T ] . Then, we have

with probability at least 1 -δ .

Proof of Lemma 15. We recall that the regret is defined as

Since a t and b t are always drawn independently from the same distribution in Algorithm 1, we only need to consider the regret of the a t part in the following proof for brevity - multiplying the result by two would yield the overall regret.

We first observe the definition of λ t in Algorithm 1: the left term ∑ t -1 s =1 Z s w s in the indicator is non-decreasing in t while the right term remains constant. This means that there exists a particular time step t ′ ∈ [ T ] dividing the time horizon into two phases: λ t = 0 for all t ≤ t ′ and λ t = 1 for all t > t ′ . Now, we proceed to examine these two phases individually.

For all rounds before or on t ′ , we can compute the expected partial regret as

where the equality holds since we have A t = { π f ⋆ ( x t ) } whenever Z t = 0 under the condition that f ⋆ ∈ F t , and thus we don't incur regret in this case. The first inequality is Lemma 8, and the second inequality holds by the definition of λ t and the condition that λ t = 0 .

On the other hand, for all rounds after t ′ , we have

where the first inequality holds by Lemma 11 (or Lemma 12 for specific function classes), the second equality is by the definition of γ t , the third inequality is by Lemma 4, and the fourth inequality holds by Lemma 9.

Putting the two parts, (8) and (9), together, we arrive at

Now we apply Lemma 4 again. The following holds with probability at least 1 -δ/ 2 ,

The above concludes the regret of the a t part. The regret of the b t can be shown in the same way. Adding them together, we conclude that

Lemma 16 (Instance-dependent regret upper bound) . For Algorithm 1, assume f ⋆ ∈ F t for all t ∈ [ T ] . Then, we have

with probability at least 1 -δ .

Proof of Lemma 16. We consider two cases. First, when

we invoke Lemma 14 and get that λ t = 0 for all t ∈ [ T ] . Hence, we have

where the first inequality is by Lemma 8 and the fact that we incur no regret when Z t = 0 since f ⋆ ∈ F t . The second inequality is by Lemma 13.

On the other hand, when the contrary of (10) holds, i.e.,

applying Lemma 15, we have

where we apply the condition (11) in the second inequality.

Lemma 17 (Query complexity) . For Algorithm 1, assume f ⋆ ∈ F t for all t ∈ [ T ] . Then, we have

with probability at least 1 -δ .

Proof of Lemma 17. We consider two cases. First, when

we can invoke Lemma 14 and get that λ t = 0 for all t ∈ [ T ] . Hence,

where the second equality is by Lemma 7, the second inequality holds as p t ( a ) is uniform for any a, b when λ t = 0 . We apply Lemma 4 and Lemma 10 to ( ∗ ) and obtain

Plugging this back, we obtain

On the other hand, when the contrary of (12) holds, i.e.,

Squaring both sides, we obtain

which leads to

We note that we always have Queries CB T ≤ T , and thus,

Hence, we complete the proof.

Having established the aforementioned lemmas, we are now able to advance towards the proof of Theorem 1.

Proof of Theorem 1. By Lemma 9 and the construction of version spaces F t in Algorithm 1, we have f ⋆ ∈ F t for all t ∈ [ T ] with probability at least 1 -δ . Then, the rest of the proof follows from Lemmas 15 to 17.

C.4 Proof of Theorem 2

In this section, we will prove the following theorem, which is stronger than Theorem 2. Theorem 5 (Lower bounds) . The following two claims hold:

for any algorithm, there exists an instance that leads to Regret CB T = Ω( √ AT ) ;

for any algorithm achieving a worse-case expected regret upper bound in the form of E [Regret CB T ] = O ( √ A · T 1 -β ) for some β > 0 , there exists an instance with gap ∆ = √ A · T -β that results in E [Regret CB T ] = Ω( A/ ∆) = Ω( √ A · T β ) and E [Queries CB T ] = Ω( A/ ∆ 2 ) = Ω( T 2 β ) .

We observe that Theorem 2 can be considered as a corollary of the above theorem when setting β = 1 / 2 .

In what follows, we will first demonstrate lower bounds in the setting of multi-armed bandits (MAB) with active queries and subsequently establish a reduction from it to contextual dueling bandits in

order to achieve these lower bounds. We start by formally defining the setting of MAB with active queries below.

Multi-armed bandits with active queries. We consider a scenario where there exist A arms. Each arm a is assumed to yield a binary reward (0 or 1), which is sampled from a Bernoulli distribution Bern (¯ r a ) , where ¯ r a denotes the mean reward associated with arm a .The arm with the highest mean reward is denoted by a ⋆ := arg max a ¯ r a . Let ∆ a := ¯ r a ⋆ -¯ r a denote the gap of arm a ∈ [ A ] . The interaction proceeds as follows: at each round t ∈ [ T ] , we need to pull an arm but can choose whether to receive the reward signal (denote this choice by Z t ). The objective is to minimize two quantities: the regret and the number of queries,

Towards the lower bounds, we will start with a bound on the KL divergence over distributions of runs under two different bandits. This result is a variant of standard results which can be found in many bandit literature (e.g., Lattimore and Szepesvári [2020]).

Lemma 18. Let I 1 and I 2 be two instances of MAB. We define p 1 and p 2 as their respective distributions over the outcomes of all pulled arms and reward signals when a query is made. Concretely, p 1 and p 2 are measuring the probability of outcomes (denoted by O ) in the following form:

where the reward r t is included only when Z t = 1 , and we added parentheses above to indicate this point. We denote Pr 1 (resp. Pr 2 ) as the reward distribution of I 1 (resp. I 2 ). We define ¯ n a = ∑ T t =1 Z t 1 { a t = a } as the number of times arm a is pulled when making a query. Then, given any algorithm A , the Kullback-Leibler divergence between p 1 and p 2 can be decomposed in the following way

Proof of Lemma 18. We define the conditional distribution

and similarly for Pr 2 . Additionally, we denote Pr A as the probability associated with algorithm A . Then, for any outcome O , we have

and we can write p 2 ( O ) in a similar manner. Hence,

where the third equality holds by the definition of Pr 1 and Pr 2 .

The following lemma establishes lower bounds for MAB with active queries. It presents a trade-off between the regret and the number of queries.

Lemma 19. Let I denote the set of all MAB instances. Assume ALG is an algorithm that achieves the following worst-case regret upper bound for some C and β :

for all I ∈ I . Then, for any MAB instance I ∈ I , the regret and the number of queries made by algorithm ALG are lower bounded:

̸

̸

where the coefficient ζ = min a min { ¯ r a , 1 -¯ r a } depends on the instance I .

̸

Proof of Lemma 19. For any MAB instance I and any arm a † , we define a corresponding MAB instance I ′ as follows. Denote ¯ r and ¯ r ′ as the mean reward of I and I ′ , respectively. For I ′ , we set the mean reward ¯ r ′ a = ¯ r a for any a = a † and ¯ r ′ a † = ¯ r a † +2∆ a † . Consequently, the optimal arm of I ′ is a † with margin ∆ a † . Let n a denote the number of times that arm a is pulled. We define the event

Hence,

Then, we have

where the last inequality is by Lemma 6. Putting the above two inequality together, we arrive at

This establishes a query lower bound for arm a † . Consequently, we have

̸

̸

and similarly,

̸

Now we can proceed with the proof of Theorem 5.

Proof of Theorem 5. We provide a reduction from the multi-armed bandits with active queries to the contextual dueling bandits. Our desired lower bound for the contextual dueling bandit setting thus follows from the above lower bound for Multi-Armed Bandits (MABs). Let ALG denote any algorithm for contextual dueling bandits.

Reduction. Since we focus on the multi-armed bandit where no context is involved, we just ignore the notation of context everywhere for brevity. We will start from an MAB instance, and then simulate a binary feedback and feed it to a dueling bandit algorithm ALG which is used to solve the original MAB instance. Particularly, consider the MAB instance with A-many actions each with an expected reward denoted as ¯ r a .

At the beginning of iteration t in the MAB instance, the learner calls the dueling algorithm ALG to generate two actions a t and b t . The learner plays a t at iteration t to receive a reward y a t ; the learner then moves to iteration t +1 to play b t , and receives reward y b t . At the end of iteration t +1 , the learner simulates a binary feedback by setting o = 1 if y a t > y b t ; o = -1 if y a t < y b t ; o being 1 or -1 uniform randomly if y a t = y b t . Then, the learner sends ( a t , b t , o ) to the dueling algorithm ALG to query for two actions which will be played at iterations t +2 and t +3 , respectively.

̸

From the dueling algorithm ALG 's perspective, given two actions a and b , we can verify that the probability of seeing label 1 is (¯ r a -¯ r b +1) / 2 . So we can just specify the link function to be ϕ ( d ) = ( d +1) / 2 . As we verified earlier, the corresponding Φ is strongly convex (Example 2). Moreover, since f ⋆ ( a, b ) = ¯ r a -¯ r b , if we define the gap of the MAB instance as ¯ ∆ := min a = a ⋆ (¯ r a ⋆ -¯ r a ) where a ⋆ := arg max i ¯ r i , then we have ¯ ∆ = ∆ in this reduction where ∆ is the definition of the gap in the dueling setting. We further note that the regret of the MAB instance is

which, by our definition of f ⋆ , is equivalent to the preference-based regret that occurred to the dueling algorithm ALG . The number of queries is clearly equivalent as well. Thus, the regret and the query complexity of the dueling algorithm ALG can be directly translated to the regret and the query complexity of the MAB instance.

Now, we are ready to prove the two claims in our statement.

Proof of the first claim. We refer the reader to Lattimore and Szepesvári [2020, Theorem 15.2] for a proof of the minimax regret lower bound of Ω( √ AT ) for the MAB. Through the reduction outlined above, that lower bound naturally extends to the dueling bandits setting, yielding Regret CB T ≥ Ω( √ AT ) (otherwise, via the above reduction, we would have achieved an approach that breaks the lower bound of MAB).

Proof of the second claim. We choose an arbitrary MAB for which ζ = min a min { ¯ r a , 1 -¯ r a } > 0 . 2 and the gaps of all arms are equal to ∆ . Invoking Lemma 19, we have

We further choose ∆ = 40 CT -β and C = A , leading to

̸

Via the reduction we have shown above, these lower bounds naturally extend to the contextual dueling bandit setting, thereby completing the proof.

C.4.1 Alternative Lower Bounds Conditioning on the Limit of Regret

In this section, we establish an analogue of Theorem 5 but under a different condition. We first introduce the concept of diminishing regret .

Definition 4. We say that an algorithm guarantees a diminishing regret if for all contextual dueling bandit instances and p > 0 , it holds that

The lower bounds under the assumption of diminishing regret guarantees are stated as follows.

Theorem 6 (Lower bounds) . The following two claims hold:

for any algorithm, there exists an instance that leads to Regret CB T ≥ Ω( √ AT ) ;

for any gap ∆ and any algorithm achieving diminishing regret, there exists an instance with gap ∆ that results in E [Regret CB T ] ≥ Ω( A/ ∆) and E [Queries CB T ] ≥ Ω( A/ ∆ 2 ) for sufficiently large T .

We should highlight that the condition of diminishing regret (Theorem 6) and the worst-case regret upper bounds (Theorems 2 and 5) are not comparable in general. However, Theorem 6 is also applicable to our algorithm (Algorithm 1) since our algorithm possesses an instance-dependent regret upper bound that is clearly diminishing.

To prove Theorem 6, we first show the following lemma, which is a variant of Lemma 19.

Lemma 20. Let I denote the set of all MAB instances. Assume ALG is an algorithm that achieves diminishing regret for all MAB instances in I , i.e., for any I ∈ I and p > 0 , it holds that

Then, for any MAB instance I ∈ I , the regret and the number of queries made by algorithm ALG are lower bounded in the following manner:

̸

̸

where the coefficient ζ := min a min { ¯ r a , 1 -¯ r a } depends on the instance I . Recall that Regret T and Queries T are defined in (13) .

̸

Proof of Lemma 20. The proof is similar to Lemma 19. For any MAB instance I ∈ I and any arm a † , we define a corresponding MAB instance I ′ as follows. Denote ¯ r and ¯ r ′ as the mean reward of I and I ′ , respectively. For I ′ , we set the mean reward ¯ r ′ a = ¯ r a for any a = a † and ¯ r ′ a † = ¯ r a † +2∆ a † . Consequently, the optimal arm of I ′ is a † with margin ∆ a † . Let n a denote the number of times that arm a is pulled. We define the event

Let p and p ′ denote the probability of I and I ′ , respectively. Then, we have

where E ∁ means the complement of event E . Hence,

Here TV denotes the total variation distance. By Lemma 18, we have

where the last inequality is by Lemma 6. Putting it all together, we arrive at

Taking the limit on both sides yields

Here the limit of (i) is clearly 0. For the limit of (ii) , we note that by the definition of diminishing regret, for any C > 0 , there exists a T ′ such that E [Regret T ] /T p ≤ C for any T > T ′ . This implies

for any p > 0 . Therefore, the limit of (ii) is also 0. Plugging these back, we obtain

This establishes a query lower bound for arm a † . Consequently, we have

̸

̸

and similarly,

̸

̸

Now, we proceed with the proof of Theorem 6.

Proof of Theorem 6. The proof of the first claim is the same as Theorem 5, so we will omit it here. Let us now focus on the proof of the second claim. By Lemma 20, for any algorithm achieving diminishing regret, the following is true for any MAB instance:

̸

̸

We choose an arbitrary MAB for which ζ ≥ 0 . 2 and the gaps of all suboptimal arms are equal to ∆ . Then, for this instance, we have

By the definition of limit, when T is large enough (exceeding a certain threshold), we have

Via the reduction we have shown in the proof of Theorem 5, these lower bounds naturally extend to the contextual dueling bandit setting, thereby completing the proof.

C.5 Proof of Theorem 3

Proof of Theorem 3. We establish the bounds for regret and the number of queries, consecutively. First, we set an arbitrary gap threshold ϵ > 0 . Since our algorithm is independent of ϵ , we can later choose any ϵ that minimizes the upper bounds.

Proof of regret. We start with the regret upper bound. By definition, we have

Since a t and b t are always drawn independently from the same distribution in Algorithm 1, we only need to consider the regret of the a t part in the following proof for brevity - multiplying the result by two would yield the overall regret.

The worst-case regret upper bound presented in Lemma 15 doesn't reply on the gap assumption and thus remains applicable in this setting. Hence, we only need to prove the instance-dependent regret upper bound. To that end, we first need an analogue of Lemma 14.

Lemma 21. Fix any ϵ > 0 . Whenever

we have λ 1 = λ 2 = · · · = λ T = 0 with probability at least 1 -δ .

Proof of Lemma 21. The proof is similar to Lemma 14 and is via contradiction. Assume the inequality holds but there exists t ′ for which λ t ′ = 1 . Without loss of generality, we assume that λ t = 0 for all t < t ′ , namely that t ′ is the first time that λ t is 1. Then by definition of λ t ′ , we have

On the other hand, we have

where the inequality is by Lemma 13. The above two inequalities contradicts with the conditions.

Towards an instance-dependent regret upper bound, we adapt the proof of Lemma 16 to this setting. We consider two cases. First, when

we invoke Lemma 21 and get that λ t = 0 for all t ∈ [ T ] . Hence, we have

where the first inequality is by Lemma 8 and the fact that we incur no regret when Z t = 0 since f ⋆ ∈ F t . The second inequality is by Lemma 13.

On the other hand, when the contrary of (14) holds, i.e.,

applying Lemma 15, we have

where we apply the condition (15) in the second inequality.

Proof of the number of queries. To show an upper bound for the number of queries, we also consider two cases. First, when

we can invoke Lemma 21 and get that λ t = 0 for all t ∈ [ T ] . Hence, similar to the proof of Lemma 17, we have

where the second inequality holds as p t ( a ) is uniform for any a, b when λ t = 0 . We apply Lemma 4 and Lemma 10 to ( ∗ ) and obtain

Plugging this back, we obtain

where the second line corresponds to the upper bound derived from the alternative case, which is shown below.

When the contrary of (16) holds, i.e.,

Squaring both sides and leveraging the inequality ( a + b ) 2 ≤ 2 a 2 +2 b 2 , we obtain

which leads to

We note that we always have Queries CB T ≤ T and thus

Minimizing on ϵ . Given that the aforementioned proofs hold for any threshold ϵ , we can select the specific value of ϵ that minimizes the upper bounds. Hence, we deduce the desired result.

C.6 Proof of Theorem 4

Proof of Theorem 4. The upper bound of the number of queries is straightforward: Algorithm 2 is simply running H instances of Algorithm 1, so the total number of queries is simply the sum of these H instances. For bounding the regret, we have

where the first inequality holds by Lemma 5, and we denote π + h ( x t,h ) = arg max a Q π e h ( x t,h , a ) in the second inequality. Then, we can plug the upper bound of Regret CB T (Theorem 1). Moreover, we need to take a union bound over all h ∈ [ H ] .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the learning problem of contextual bandits and imitation learning, where the learner lacks direct knowledge of the executed actions's reward (feedback), instead, the learner is only able to request the expert at each round to compare two actions. \n\n[Interaction Protocol] The interaction between the learner and the environment proceeds in rounds with $T$ being the total number of interactions. In each round $t$, the learner first receives the context $x_t$ (which is drawn adversarially), decide whether to send request to the expert, and selects the actions (a pair of actions in the contextual bandit setting as shown in Algorithm 1). \n\n[Preference, Request and General Function Class] For the request associated with a pair of actions $(a_t,b_t)$, the feedback $y_t \\in \\{ -1,+1}$ indicates either $a_t$ or $b_t$ is better, which follows an unknown preference function $f^\\star$ (defined in Line 122). The learner has access to a general function class $\\mathcal{F}$ where $f^\\star \\in \\mathcal{F}$, as stated in Assumption 1. \n\n[Goal] The performance of the learner is measured by (see Line 143 for the detailed definition): 1) the regret she suffered, that is, the difference between her total loss, and that of the optimal action; 2) the number of requests sent by the learner. \n\n[Result of Contextual Bandits] With specific link function and online regression oracle defined in Assumption 2, Theorem 1 ensures that the regret is bounded by $\\widetilde{\\mathcal{O}}(\\min\\{ \\sqrt{T}, \\frac{d}{\\Delta} \\})$ where $d$ is some sort of complexity measurement (Eluder dimension) and $\\Delta$ is the uniform gap (the minimal gap between the best and the second best action over all the context) defined in Assumption 3. The number of queries is also bounded as shown in Theorem 1. This result further matches the lower bound up to some factors (see Theorem 2 for more details). \n\n[Imitation Result] Theorem 4 states the result and the details are deferred to Appendix, which I don't have much time to check carefully.\n\nSTRENGTHS:\nThe paper is clearly-written, well-organized and rigorous. \n\n1. The proof is self-contained. I haven't observed any mistakes in the lemmas I skimmed. \n2. The notations, together with their meanings, are well explained. T\n3. The definitions and assumptions are carefully separated.\n\nWEAKNESSES:\n1. Considering the dueling bandit problem, a special case of the problem instance studied , does the regret bound stated in Theorem 1 matches the optimal regret bound for dueling bandits? I am not quite certain about the scale of $dim_{E}(\\mathcal{F}, \\frac{\\Delta}{2A^2})$ in this case.\n2. The comparisons between the Theorem 1 and the results of Saha and Krishnamurthy [2022], and Foster et al. [2020] are not very detailed. What's regret bound of the naive conversion of ADACB into regret minimization problem, and how does it relate to this work?\n\nQUESTIONS:\nMy questions is raised in the weakness section. I am willing to re-evaluate the score if the questions are answered properly.\n\nLIMITATIONS:\nThis work is pure theoretical, and does not have any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the contextual bandit and imitation learning problem with preference-based feedback. The authors propose an oracle-based contextual bandit algorithm, which attains both worst-case and instance-dependent regret bounds. Besides, the algorithm has an instance-dependent guarantee on the querying numbers of the preference-based information. Furthermore, the proposed bandit algorithm is extended to the imitation learning setting with provable guarantees.\n\nSTRENGTHS:\n- the proposed method has strong theoretical guarantees on the regret (both worst-case and instance-dependent bound) and query complexity. Although the oracle-based algorithm proposed shares similar techniques with MinMaxDB [Saha and Krishnamurthy, 2022] and AdaCB [Foster et al., 2020], the authors provide enough discussion to highlight the difference.\n- lower bounds are provided to justify the upper bounds on regret, and query complexity is tight up to logarithmic factors\n- the paper is well-structured and written\n\nWEAKNESSES:\n- about the practical implementation of the proposed method: one of my main concerns about the paper is from the practical side. Similar to the oracle-based algorithm for the standard contextual bandit problem (e.g., SquareCB [Foster et al. 2022]), the proposed method is established on an online regression solver with regret guarantees. However, I'm not sure to what extent such an online regression solver can be obtained with the preference-based feedback model. For instance, as shown in example 1, $f(x, a,b) = r(a,x)-r(x,b)$, the function $f(\\cdot)$ is not convex even $r:\\mathcal{X}\\times\\mathcal{A}\\rightarrow[0,1]$ is a convex function, and the algorithm developed for online convex optimization is not applicable. I think it would be beneficial if the authors could provide some concrete examples (for example, the reward function has a linear structure?) that the online regression oracle is available. \n\n-  about the instance-dependent bound: the proposed instance-dependent regret bound as an $O(\\Upsilon^2)$ dependence on the regret of the oracle and an  $O(\\Upsilon^3)$ on the query complexity. There seems still some room for improvement. In the finite function space case, AdaCB attains an $O(\\log \\vert\\mathcal{F}\\vert/\\Delta)$ bound for a standard contextual bandit problem, but the result obtained in this paper implies an $O(\\log^2 \\vert\\mathcal{F}\\vert/\\Delta)$ regret bound.\n\nQUESTIONS:\n- could you provide concrete examples of the online regression oracle for the preference-based feedback model? It would be even better if the author could provide more detailed discussions on to which extent such an online regression solver can be established.\n\n- could you provide more discussion on the tightness of the instance-dependent bound, especially on the dependence of $\\Upsilon$?\n\n- The expert policy $\\pi_e$ is not formally defined. Does $\\pi_e$ refer to the policy that can maximize the value function? I am confused by the claim, \"our algorithm not only competes with the expert policy but can also surpass it to some extent\" in line 343. What is the formal definition of \"surpass.\" Do you mean the regret would go negative due to the term $Adv_T$? However, it is unclear to me when the negative term is large enough to cancel the $O(\\sqrt{T}, A/\\Delta)$ term.\n\nLIMITATIONS:\nThe paper has discussed the limitation and potential future work in the conclusion. Another issue is that it imposes a realizable assumption for $f^\\star$. It is unclear whether extending the analysis for standard contextual bandit (Section 5 in [Foster et al., ICML 2020]) to the contextual dueling bandit setting is possible.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives “best-of-both-worlds” results for an imitation-learning problem in contextual bandits and MDP settings. With small orthogonal changes to assumptions, the algorithms primarily improve over prior work by considering instance-optimal bounds both in regret and queries, and require only ordinal preference feedback rather than explicit rewards (similar to the “dueling bandits“ literature).\n\nSTRENGTHS:\n- The paper is easy to read, the algorithms and notation are well-explained, and the results are appropriately contextualized in prior work.\n- The examples given for the functions in the model are quite useful for grounding the problem in more concrete applications. Related work is discussed thoroughly.\n- Conceptually, the model draws nice connections between contextual bandits and modern topics in finetuning models (e.g. LLMs) from preference feedback, where the emphasis on “instance-optimal” style results is particularly well-motivated.\n\nWEAKNESSES:\n- While the application of techniques from online reinforcement learning to obtain the instance-optimal bounds in this setting is clever, it is unclear how much of this follows directly vs what technical innovation is required. It would be helpful to highlight the methodological contributions used.\n- Given the applications discussed, it would be beneficial to give experimental results for preference finetuning (even in a toy setting) to demonstrate the importance of instance-optimality in practice.\n- While the instance-optimal rates seem reasonable, it would be nice to include (partially) matching lower bounds for some results, or discuss barriers to obtaining such results.\n\nQUESTIONS:\n- Can the rates on $d$ or $\\Delta$ be shown to be asymptotically tight for either queries or regret? \n- What does the notation $P_t[a_t, b_t]$ on line 146 refer to?\n\nLIMITATIONS:\n- Connections to prior RL work which makes use of eluder dimension could be discussed in greater detail.\n- Some hyperlinks are broken in the PDF.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops the provably efficient algorithms AURORA and AURORAE, which are able to achieve the optimal regret bound under contextual dueling bandit setting, and imitation learning respectively, at the same time minimizing query complexity. The key idea behind is that the algorithm only makes a query when the algorithm is very uncertain about the optimal action ($Z_t 1_{|A_t| > 1}$). The algorithm decides the sampling distribution of action pairs to make a query by considering whether the estimated cumulative regret exceeds the carefully designed threshold. If it does not exceed, the algorithm does exploration and sample action pairs from the uniform distribution. If it exceeds, the algorithm uses a technique similar to inverse gap weighting to achieve better balance between exploration and exploitation. For imitation setting with horizon H, the algorithm treats MDP as a concatenation of H contextual bandits and runs AURORAE, which is a stack of multiple AURORA instances.\n\nSTRENGTHS:\nThis work is original and well-motivated. It is crucial to design an online learning algorithm that achieves optimal regret while using minimal query complexity. Although I did not get a chance to read the complete proofs in the supplementary material carefully, given the discussion of intuition, all technical results seem reasonable to me. \n\nThis paper is well presented and is a pleasure to read. An example for illustration follows every definition. All materials are well organized in a logical manner.\n\nWEAKNESSES:\nI have several concerns regarding the proposed algorithms. First, P5 I 5, the computational complexity for the candidate arm set might be very large, even if F is assumed to be a d-dimensional linear class. The computational complexity might be $O(dT\\log(T)|A|)$. Also, in reality, F might be very complex, which might even worsen the computational complexity. Can we use a simple function class F for approximation while still achieving a similar regret bound?\n\nQUESTIONS:\nPlease see the review in weaknesses.\n\nLIMITATIONS:\nThe authors address their limitations of not having any experiments on real data or simulations. I believe the work will be much more convincing if the theoretical bounds are supported by experiment results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}