Title: Borda Regret Minimization for Generalized Linear Dueling Bandits
Venue: NeurIPS 2023
Accepted: no
Keywords: Borda score

dueling bandit

generalized linear model

linear bandit

regret minimization
Abstract: Dueling bandits are widely used to model preferential feedback prevalent in many applications such as recommendation systems and ranking. In this paper, we study the Borda regret minimization problem for dueling bandits, which aims to identify the item with the highest Borda score while minimizing the cumulative regret. We propose a rich class of generalized linear dueling bandit models, which cover many existing models. We first prove a regret lower bound of order $\Omega(d^{2/3} T^{2/3})$ for the Borda regret minimization problem, where $d$ is the dimension of contextual vectors and $T$ is the time horizon. To attain this lower bound, we propose an explore-then-commit type algorithm for the stochastic setting, which has a nearly matching regret upper bound $\tilde{O}(d^{2/3} T^{2/3})$. We also propose an EXP3-type algorithm for the adversarial setting, where the underlying model parameter can change at each round. Our algorithm achieves an $\tilde{O}(d^{2/3} T^{2/3})$ regret, which is also optimal. Empirical evaluations on both synthetic data and a simulated real-world environment are conducted to corroborate our theoretical analysis.

1

Abstract

Dueling bandits are widely used to model preferential feedback prevalent in many applications such as recommendation systems and ranking. In this paper, we study the Borda regret minimization problem for dueling bandits, which aims to identify the item with the highest Borda score while minimizing the cumulative regret. We propose a rich class of generalized linear dueling bandit models, which cover many existing models. We first prove a regret lower bound of order Ω( d 2 / 3 T 2 / 3 ) for the Borda regret minimization problem, where d is the dimension of contextual vectors and T is the time horizon. To attain this lower bound, we propose an explorethen-commit type algorithm for the stochastic setting, which has a nearly matching regret upper bound ˜ O ( d 2 / 3 T 2 / 3 ) . We also propose an EXP3-type algorithm for the adversarial setting, where the underlying model parameter can change at each round. Our algorithm achieves an ˜ O ( d 2 / 3 T 2 / 3 ) regret, which is also optimal. Empirical evaluations on both synthetic data and a simulated real-world environment are conducted to corroborate our theoretical analysis.

1 Introduction

Multi-armed bandits (MAB) (Lattimore and Szepesvári, 2020) is an interactive game where at each round, an agent chooses an arm to pull and receives a noisy reward as feedback. In contrast to numerical feedback considered in classic MAB settings, preferential feedback is more natural in various online learning tasks including information retrieval Yue and Joachims (2009), recommendation systems Sui and Burdick (2014), ranking Minka et al. (2018), crowdsourcing Chen et al. (2013), etc. Moreover, numerical feedback is also more difficult to gauge and prone to errors in many real-world applications. For example, when provided with items to shop or movies to watch, it is more natural for a customer to pick a preferred one than scoring the options. This motivates Dueling Bandits (Yue and Joachims, 2009), where the agent repeatedly pulls two arms at a time and is provided with feedback being the binary outcome of 'duels' between the two arms.

In dueling bandits problems, the outcome of duels is commonly modeled as Bernoulli random variables due to their binary nature. At each round, suppose the agent chooses to compare arm i and j , then the binary feedback is assumed to be sampled independently from a Bernoulli distribution. For a dueling bandits instance with K arms, the probabilistic model of the instance can be fully characterized by a K × K preference probability matrix with each entry being: p i,j = P ( arm i is chosen over arm j ) .

In a broader range of applications such as ranking, 'arms' are often referred to as 'items'. We will use these two terms interchangeably in the rest of this paper. One central goal of dueling bandits

is to devise a strategy to identify the 'optimal' item as quickly as possible, measured by either sample complexity or cumulative regret. However, the notion of optimality for dueling bandits is way

harder to define than for multi-armed bandits. The latter can simply define the arm with the highest numerical feedback as the optimal arm, while for dueling bandits there is no obvious definition solely

dependent on

{

p

i,j

|

i, j

∈

[

K

}

]

.

The first few works on dueling bandits imposed strong assumptions on p . For example, Yue et al. 38

i,j (2012) assumed that there exists a true ranking that is coherent among all items, and the preference 39 probabilities must satisfy both strong stochastic transitivity (SST) and stochastic triangle inequality 40 (STI). While relaxations like weak stochastic transitivity (Falahatgar et al., 2018) or relaxed stochastic 41 transitivity (Yue and Joachims, 2011) exist, they typically still assume the true ranking exists and the 42 preference probabilities are consistent, i.e., p i,j > 1 2 if and only if i is ranked higher than j . In reality, 43 the existence of such coherent ranking aligned with item preferences is rarely the case. For example, 44 p i,j may be interpreted as the probability of one basketball team i beating another team j , and there 45 can be a circle among the match advantage relations. 46

47

48

49

50

51

52

53

In this paper, we do not assume such coherent ranking exists and solely rely on the Borda score based on preference probabilities. The Borda score B ( i ) of an item i is the probability that it is preferred when compared with another random item, namely B ( i ) := 1 K -1 ∑ j = i p i,j . The item with the highest Borda score is called the Borda winner . The Borda winner is intuitively appealing and always well-defined for any set of preferential probabilities. The Borda score also does not require the problem instance to obey any consistency or transitivity, and it is considered one of the most general criteria.

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

̸

To identify the Borda winner, estimations of the Borda scores are needed. Since estimating the Borda score for one item requires comparing it with every other items, the sample complexity is prohibitively high when there are numerous items. On the other hand, in many real-world applications, the agent has access to side information that can assist the evaluation of p i,j . For instance, an e-commerce item carries its category as well as many other attributes, and the user might have a preference for a certain category (Wang et al., 2018). For a movie, the genre and the plot as well as the directors and actors can also be taken into consideration when making choices (Liu et al., 2017).

Based on the above motivation, we consider Generalized Linear Dueling Bandits . At each round, the agent selects two items from a finite set of items and receives a comparison result of the preferred item. The comparisons depend on known intrinsic contexts/features associated with each pair of items. The contexts can be obtained from upstream tasks, such as topic modeling (Zhu et al., 2012) or embedding (Vasile et al., 2016). Our goal is to adaptively select items and minimize the regret with respect to the optimal item (i.e., Borda winner). Our main contributions are summarized as follows:

We show a hardness result regarding the Borda regret minimization for the (generalized) linear model. We prove a worst-case regret lower bound Ω( d 2 / 3 T 2 / 3 ) for our dueling bandit model, showing that even in the stochastic setting, minimizing the Borda regret is difficult. The construction and proof of the lower bound are new and might be of independent interest.

We propose an explore-then-commit type algorithm under the stochastic setting, which can achieve a nearly matching upper bound ˜ O ( d 2 / 3 T 2 / 3 ) . When the number of items K is small, the algorithm can also be configured to achieve a smaller regret ˜ O ( ( d log K ) 1 / 3 T 2 / 3 ) .

We propose an EXP3 type algorithm for linear dueling bandits under the adversarial setting, which can achieve a nearly matching upper bound ˜ O ( ( d log K ) 1 / 3 T 2 / 3 ) .

We conduct empirical studies to verify the correctness of our theoretical claims. Under both synthetic and real-world data settings, our algorithms can outperform all the baselines in terms of cumulative regret.

Notation In this paper, we use normal letters to denote scalars, lowercase bold letters to denote vectors, and uppercase bold letters to denote matrices. For a vector x , ∥ x ∥ denotes its ℓ 2 -norm. The weighted ℓ 2 -norm associated with a positive-definite matrix A is defined as ∥ x ∥ A = √ x ⊤ Ax . The minimum eigenvalue of a matrix A is written as λ min ( A ) . We use standard asymptotic notations including O ( · ) , Ω( · ) , Θ( · ) , and ˜ O ( · ) , ˜ Ω( · ) , ˜ Θ( · ) will hide logarithmic factors. For a positive integer N , [ N ] := { 1 , 2 , . . . , N } .

2 Related Work

Multi-armed and Contextual Bandits Multi-armed bandit is a problem of identifying the best 86 choice in a sequential decision-making system. It has been studied in numerous ways with a wide 87 range of applications (Even-Dar et al., 2002; Lai et al., 1985; Kuleshov and Precup, 2014). Contextual 88 linear bandit is a special type of bandit problem where the agent is provided with side information, i.e., 89 contexts, and rewards are assumed to have a linear structure. Various algorithms (Rusmevichientong 90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

and Tsitsiklis, 2010; Filippi et al., 2010; Abbasi-Yadkori et al., 2011; Li et al., 2017; Jun et al., 2017) have been proposed to utilize this contextual information.

Dueling Bandits and Its Performance Metrics Dueling bandits is a variant of MAB with preferential feedback (Yue et al., 2012; Zoghi et al., 2014a, 2015). A comprehensive survey can be found at Bengs et al. (2021). As discussed previously, the probabilistic structure of a dueling bandits problem is governed by the preference probabilities, over which an optimal item needs to be defined. Optimality under the Borda score criteria has been adopted by several previous works (Jamieson et al., 2015; Falahatgar et al., 2017a; Heckel et al., 2018; Saha et al., 2021a). The most relevant work to ours is Saha et al. (2021a), where they studied the problem of regret minimization for adversarial dueling bandits and proved a T -round Borda regret upper bound ˜ O ( K 1 / 3 T 2 / 3 ) . They also provide an Ω( K 1 / 3 T 2 / 3 ) lower bound for stationary dueling bandits using Borda regret.

̸

Apart from the Borda score, Copeland score is also a widely used criteria (Urvoy et al., 2013; Zoghi et al., 2015, 2014b; Wu and Liu, 2016; Komiyama et al., 2016). It is defined as C ( i ) := 1 K -1 ∑ j = i 1 { p i,j > 1 / 2 } . A Copeland winner is the item that beats the most number of other items. It can be viewed as a 'thresholded' version of Borda winner. In addition to Borda and Copeland winners, optimality notions such as a von Neumann winner were also studied in Ramamohan et al. (2016); Dudík et al. (2015); Balsubramani et al. (2016).

Another line of work focuses on identifying the optimal item or the total ranking, assuming the preference probabilities are consistent. Common consistency conditions include Strong Stochastic Transitivity (Yue et al., 2012; Falahatgar et al., 2017a,b), Weak Stochastic Transitivity (Falahatgar et al., 2018; Ren et al., 2019; Wu et al., 2022; Lou et al., 2022), Relaxed Stochastic Transitivity (Yue and Joachims, 2011) and Stochastic Triangle Inequality. Sometimes the aforementioned transitivity can also be implied by some structured models like the Bradley-Terry model. We emphasize that these consistency conditions are not assumed or implicitly implied in our setting.

Contextual Dueling Bandits In Dudík et al. (2015), contextual information is incorporated in the dueling bandits framework. Later, Saha (2021) studied a structured contextual dueling bandits setting where each item i has its own contextual vector x i (sometimes called Linear Stochastic Transitivity). Each item then has an intrinsic score v i equal to the linear product of an unknown parameter vector θ ∗ and its contextual vector x i . The preference probability between two items i and j is assumed to be µ ( v i -v j ) where µ ( · ) is the logistic function. These intrinsic scores of items naturally define a ranking over items. The regret is also computed as the gap between the scores of pulled items and the best item. While in this paper, we assume that the contextual vectors are associated with item pairs and define regret on the Borda score. In Section A.1, we provide a more detailed discussion showing that the setting considered in Saha (2021) can be viewed as a special case of our model.

3 Backgrounds and Preliminaries

3.1 Problem Setting

We first consider the stochastic preferential feedback model with K items in the fixed time horizon setting. We denote the item set by [ K ] and let T be the total number of rounds. At each round t , the agent can pick any pair of items ( i t , j t ) to compare and receive stochastic feedback about whether item i t is preferred over item j t , (denoted by i t ≻ j t ). We denote the probability of seeing the event i ≻ j as p i,j ∈ [0 , 1] . Naturally, we assume p i,j + p j,i = 1 , and p i,i = 1 / 2 .

In this paper, we are concerned with the generalized linear model (GLM), where there is assumed to exist an unknown parameter θ ∗ ∈ R d , and each pair of items ( i, j ) has its own known contextual/feature vector ϕ i,j ∈ R d with ∥ ϕ i,j ∥ ≤ 1 . There is also a fixed known link function (sometimes called comparison function) µ ( · ) that is monotonically increasing and satisfies µ ( x ) + µ ( -x ) = 1 , e.g. a linear function or the logistic function µ ( x ) = 1 / (1 + e -x ) . The preference probability is defined as p i,j = µ ( ϕ ⊤ i,j θ ∗ ) . At each round, denote r t = 1 { i t ≻ j t } , then we have

Then our model can also be written as 138

where the noises { ϵ t } t ∈ [ T ] are zero-mean, 1 -sub-Gaussian and assumed independent from each other. 139 Note that, given the constraint p i,j + p j,i = 1 , it is implied that ϕ i,j = -ϕ j,i for any i ∈ [ K ] , j ∈ [ K ] . 140

141

142

143

144

145

146

147

148

149

The agent's goal is to maximize the cumulative Borda score. The (slightly modified 1 ) Borda score of item i is defined as B ( i ) = 1 K ∑ K j =1 p i,j , and the Borda winner is defined as i ∗ = argmax i ∈ [ K ] B ( i ) . The problem of merely identifying the Borda winner was deemed trivial (Zoghi et al., 2014a; Busa-Fekete et al., 2018) because for a fixed item i , uniformly random sampling j and receiving feedback r i,j = Bernoulli( p i,j ) yield a Bernoulli random variable with its expectation being the Borda score B ( i ) . This so-called Borda reduction trick makes identifying the Borda winner as easy as the best-arm identification for K -armed bandits. Moreover, if the regret is defined as Regret( T ) = ∑ T t =1 ( B ( i ∗ ) -B ( i t )) , then any optimal algorithms for multi-arm bandits can achieve ˜ O ( √ T ) regret.

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

However, the above definition of regret does not respect the fact that a pair of items are selected at each round. When the agent chooses two items to compare, it is natural to define the regret so that both items contribute equally. A commonly used regret, e.g., in Saha et al. (2021a), has the following form:

where the regret is defined as the sum of the sub-optimality of both selected arms. Sub-optimality is measured by the gap between the Borda scores of the compared items and the Borda winner. This form of regret deems any classical multi-arm bandit algorithm with Borda reduction vacuous because taking j t into consideration will invoke Θ( T ) regret.

Adversarial Setting Saha et al. (2021b) considered an adversarial setting for the multi-armed case, where at each round t , the comparison follows a potentially different probability model, denoted by { p t i,j } i,j ∈ [ K ] . In this paper, we consider its contextual counterpart. Formally, we assume there is an underlying parameter θ ∗ t , and at round t , the preference probability is defined as p t i,j = µ ( ϕ ⊤ i,j θ ∗ t ) .

The Borda score of item i ∈ [ K ] at round t is defined as B t ( i ) = 1 K ∑ K j =1 p t i,j , and the Borda winner at round T is defined as i ∗ = argmax i ∈ [ K ] ∑ T t =1 B t ( i ) . The T -round regret is thus defined as Regret( T ) = ∑ T t =1 ( 2 B t ( i ∗ ) -B t ( i t ) -B t ( j t ) ) .

3.2 Assumptions

In this section, we present the assumptions required for establishing theoretical guarantees. Due to the fact that the analysis technique is largely extracted from Li et al. (2017), we follow them to make assumptions to enable regret minimization for generalized linear dueling bandits.

We make a regularity assumption about the distribution of the contextual vectors:

Assumption 1. There exists a constant λ 0 > 0 such that λ min ( 1 K 2 ∑ K i =1 ∑ K j =1 ϕ i,j ϕ ⊤ i,j ) ≥ λ 0 .

This assumption is only utilized to initialize the design matrix V τ = ∑ τ t =1 ϕ i t ,j t ϕ ⊤ i t ,j t so that the minimum eigenvalue is large enough. We follow Li et al. (2017) to deem λ 0 as a constant.

We also need the following assumption regarding the link function µ ( · ) :

Assumption 2. Let ˙ µ be the first-order derivative of µ . We have κ := inf ∥ x ∥≤ 1 , ∥ θ -θ ∗ ∥≤ 1 ˙ µ ( x ⊤ θ ) > 0 .

Assuming κ > 0 is necessary to ensure the maximum log-likelihood estimator can converge to the true parameter θ ∗ (Li et al., 2017, Section 3). This type of assumption is commonly made in previous works for generalized linear models (Filippi et al., 2010; Li et al., 2017; Faury et al., 2020).

Another common assumption is regarding the continuity and smoothness of the link function. 179

̸

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

Assumption 3. µ is twice differentiable. Its first and second-order derivatives are upper-bounded by constants L µ and M µ respectively.

This is a very mild assumption. For example, it is easy to verify that the logistic link function satisfies Assumption 3 with L µ = M µ = 1 / 4 .

4 The Hardness Result

'good'

{

'bad'

{





















1

2

· · ·

1

2

.

.

.

.

.

.

.

.

.

1

2

· · ·

1

2

3

4

+

⟨

ϕ

i,j

,

θ

⟩

1

4

+

⟨

ϕ

j,i

,

θ

⟩

1

2

· · ·

1

2

.

.

.

.

.

.

.

.

.

1

2

· · ·

1

2





















3

4

+

⟨

bit

(0)

,

θ

⟩

3

4

+

⟨

bit

(0)

,

θ

⟩

· · ·

3

4

+

⟨

bit

(0)

,

θ

⟩

3

4

+

⟨

bit

(1)

,

θ

⟩

3

4

+

⟨

bit

(1)

,

θ

⟩

· · ·

3

4

+

⟨

bit

(1)

,

θ

⟩

.

.

.

.

.

.

.

.

.

.

.

.

3

4

+

⟨

bit

(2

d

-

1)

,

θ

⟩

3

4

+

⟨

bit

(2

d

-

1)

,

θ

⟩ · · ·

3

4

+

⟨

bit

(2

d

-

1)

,

θ

⟩

This section presents Theorem 4, a worst-case regret lower bound for the stochastic linear dueling bandits. The proof of Theorem 4 relies on a class of hard instances, as shown in Figure 1. We show that any algorithm will incur a certain amount of regret when applied to this hard instance class. The constructed hard instances follow a stochastic linear model, which is a sub-class of the generalized linear model. Saha et al. (2021b) first proposed a similar construction for finite many arms with no contexts. Our construction is for a contextual setting and the proof of the lower bound takes a rather different route.

For any d > 0 , we construct the class of hard instances as follows. An instance is specified by a vector θ ∈ {-∆ , +∆ } d . The instance contains 2 d +1 items (indexed from 0 to 2 d +1 -1 ). The preference probability for an instance is defined by p θ i,j as:

and the d -dimensional feature vectors ϕ i,j are given by

where bit ( · ) is the (shifted) bit representation of non-negative integers, i.e., suppose x has the binary representation x = b 0 × 2 0 + b 1 × 2 1 + · · · + b d -1 × 2 d -1 , then

Note that bit ( · ) ∈ {-1 , +1 } d , and that ϕ i,j = -ϕ j,i is satisfied. The definition of p θ i,j can be slightly tweaked to fit exactly the model described in Section 3 (see Remark 11 in Appendix).

Some calculation shows that the Borda scores of the 2 d +1 items are:

Intuitively, the former half of items (those indexed from 0 to 2 d -1 ) are 'good' items (one among them is optimal, others are nearly optimal), while the latter half of items are 'bad' items. Under such hard instances, every time one of the two pulled items is a 'bad' item, then a one-step regret

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

B θ ( i ∗ ) -B θ ( i ) ≥ 1 / 4 is incurred. To minimize regret, we should thus try to avoid pulling 'bad' items. However, in order to identify the best item among all 'good' items, comparisons between 'good' and 'bad' items are necessary. The reason is simply that comparisons between 'good' items give no information about the Borda scores as the comparison probabilities are p θ i,j = 1 2 for all i, j < 2 d . Hence, any algorithm that can decently distinguish among the 'good' items has to pull 'bad' ones for a fair amount of times, and large regret is thus incurred. A similar observation is also made by Saha et al. (2021a).

This specific construction emphasizes the intrinsic hardness of Borda regret minimization: to differentiate the best item from its close competitors, the algorithm must query the bad items to gain information.

Formally, this class of hard instances leads to the following regret lower bound for both stochastic and adversarial settings:

Theorem 4. For any algorithm A , there exists a hard instance { p θ i,j } with T > 4 d 2 , such that A will incur expected regret at least Ω( d 2 / 3 T 2 / 3 ) .

The construction of this hard instance for linear dueling bandits is inspired by the worst-case lower bound for the stochastic linear bandit (Dani et al., 2008), which has the order Ω( d √ T ) , while ours is Ω( d 2 / 3 T 2 / 3 ) . The difference is that for the linear or multi-armed stochastic bandit, eliminating bad arms can make further exploration less expensive. But in our case, any amount of exploration will not reduce the cost of further exploration. This essentially means that exploration and exploitation must be separate, which is also supported by the fact that a simple explore-then-commit algorithm shown in Section 5 can be nearly optimal.

5 Stochastic Contextual Dueling Bandit

5.1 Algorithm Description

Algorithm 1 BETC-GLM

1: Input: time horizon T , number of items K , feature dimension d , feature vectors ϕ i,j for i ∈ [ K ] , j ∈ [ K ] , exploration rounds τ , error tolerance ϵ , failure probability δ .

2: for t = 1 , 2 , . . . , τ do

3: sample i t ∼ Uniform([ K ]) , j t ∼ Uniform([ K ])

4: query pair ( i t , j t ) and receive feedback r t

5: end for

6: Find the G-optimal design π ( i, j ) based on ϕ i,j for i ∈ [ K ] , j ∈ [ K ]

8: for i ∈ [ K ] , j ∈ [ K ] , s ∈ [ N ( i, j )] do

9: set t ← t +1 , set ( i t , j t ) = ( i, j )

10: query pair ( i t , j t ) and receive feedback r t

11: end for

12: Calculate the empirical MLE estimator ̂ θ τ + N based on all τ + N samples via (2)

13: Estimate the Borda score for each item:

14: Keep querying ( ̂ i, ̂ i ) for the rest of the time.

We propose an algorithm named Borda Explore-Then-Commit for Generalized Linear Models (BETC-GLM), presented in Algorithm 1. Our algorithm is inspired by the algorithm for generalized linear models proposed by Li et al. (2017).

At the high level, Algorithm 1 can be divided into two phases: the exploration phase (Line 2-11) and the exploitation phase (Line 12-14). The exploration phase ensures that the MLE estimator ̂ θ is accurate enough so that the estimated Borda score is within ˜ O ( ϵ ) -range of the true Borda score

(ignoring other quantities). Then the exploitation phase simply chooses the empirical Borda winner 233 to incur small regret. 234

235

236

237

During the exploration phase, the algorithm first performs 'pure exploration' (Line 2-5), which can be seen as an initialization step for the algorithm. The purpose of this step is to ensure the design matrix V τ + N = ∑ τ + N t =1 ϕ i t ,j t ϕ ⊤ i t ,j t is positive definite.

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

After that, the algorithm will perform the 'designed exploration'. Line 6 will find the G-optimal design, which minimizes the objective function g ( π ) = max i,j ∥ ϕ i,j ∥ 2 V ( π ) -1 , where V ( π ) := ∑ i,j π ( i, j ) ϕ i,j ϕ ⊤ i,j . The G-optimal design π ∗ ( · ) satisfies ∥ ϕ i,j ∥ 2 V ( π ∗ ) -1 ≤ d , and can be efficiently approximated by the Frank-Wolfe algorithm (See Remark 8 for a detailed discussion). Then the algorithm will follow π ( · ) found at Line 6 to determine how many samples (Line 7) are needed. At Line 8-11, there are in total N = ∑ K i =1 ∑ K j =1 N ( i, j ) samples queried, and the algorithm shall index them by t = τ +1 , τ +2 , . . . , τ + N .

At Line 12, the algorithm collects all the τ + N samples and performs the maximum likelihood estimation (MLE). For the generalized linear model, the MLE estimator ̂ θ τ + N satisfies:

or equivalently, it can be determined by solving a strongly concave optimization problem:

where ˙ m ( · ) = µ ( · ) . For the logistic link function, m ( x ) = log(1 + e x ) . As a special case of our generalized linear model, the linear model has a closed-form solution for (2). For example, if µ ( x ) = 1 2 + x , i.e. p i,j = 1 2 + ϕ ⊤ i,j θ ∗ , then (2) becomes:

where V τ + N = ∑ τ + N t =1 ϕ i t ,j t ϕ ⊤ i t ,j t .

After the MLE estimator is obtained, Line 13 will calculate the estimated Borda score ̂ B ( i ) for each item based on ̂ θ τ + N , and pick the empirically best one.

5.2 A Matching Regret Upper Bound

Algorithm 1 can be configured to tightly match the worst-case lower bound. The configuration and performance are described as follows:

Theorem 5. Suppose Assumption 1-3 hold and T = Ω( d 2 ) . For any δ > 0 , if we set τ = C 4 λ -2 0 ( d +log(1 /δ )) ( C 4 is a universal constant) and ϵ = d 1 / 6 T -1 / 3 , then with probability at least 1 -2 δ , Algorithm 1 will incur regret bounded by:

By setting δ = T -1 , the expected regret is bounded as ˜ O ( κ -1 d 2 / 3 T 2 / 3 ) .

For linear bandit models, such as the hard-to-learn instances in Section 4, κ is a universal constant. Therefore, Theorem 5 tightly matches the lower bound in Theorem 4, up to logarithmic factors.

Remark 6 (Regret for Fewer Arms) . In typical scenarios, the number of items K is not exponentially large in the dimension d . In this case, we can choose a different parameter set of τ and ϵ such that Algorithm 1 can achieve a smaller regret bound ˜ O ( κ -1 ( d log K ) 1 / 3 T 2 / 3 ) with smaller dependence on the dimension d . See Theorem 10 in Appendix A.2.

Remark 7 (Regret for Infinitely Many Arms) . In most practical scenarios of dueling bandits, it is adequate to consider a finite number K of items (e.g., ranking items). Nonetheless, BETC-GLM

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

can be easily adapted to accommodate infinitely many arms in terms of regret. We can construct a covering over all ϕ and perform optimal design and exploration on the covering set. The resulting i,j regret will be the same as our upper bound, i.e., ˜ O ( d 2 / 3 T 2 / 3 ) up to some error caused by the epsilon net argument.

Remark 8 (Approximate G-optimal Design) . Algorithm 1 assumes an exact G-optimal design π is obtained. In the experiments, we use the Frank-Wolfe algorithm to solve the constraint optimization problem (See Algorithm 5, Appendix G.3). To find a policy π such that g ( π ) ≤ (1 + ε ) g ( π ∗ ) , roughly O ( d/ε ) optimization steps are needed. Such a near-optimal design will introduce a factor of (1 + ε ) 1 / 3 into the upper bounds.

6 Adversarial Contextual Dueling Bandit

This section addresses Borda regret minimization under the adversarial setting. As we introduced in Section 3.1, the unknown parameter θ t can vary for each round t , while the contextual vectors ϕ i,j are fixed.

Our proposed algorithm, BEXP3, is designed for the contextual linear model. Formally, at round t and given pair ( i, j ) , we have p t i,j = 1 2 + ⟨ ϕ i,j , θ ∗ t ⟩ .

6.1 Algorithm Description

Algorithm 2 BEXP3

1: Input: time horizon T , number of items K , feature dimension d , feature vectors ϕ i,j for i ∈ [ K ] , j ∈ [ K ] , learning rate η , exploration parameter γ .

2: Initialize: q 1 ( i ) = 1 K .

3: for t = 1 , . . . , T do

4: Sample items i t ∼ q t , j t ∼ q t .

5: Query pair ( i t , j t ) and receive feedback r t

6: Calculate Q t = ∑ i ∈ [ K ] ∑ j ∈ [ K ] q t ( i ) q t ( j ) ϕ i,j ϕ ⊤ i,j , ̂ θ t = Q -1 t ϕ i t ,j t r t .

7: Calculate the (shifted) Borda score estimates ̂ B t ( i ) = ⟨ 1 K ∑ j ∈ [ K ] ϕ i,j , ̂ θ t ⟩ .

8: Update for all i ∈ [ K ] , set

9: end for

Algorithm 2 is adapted from the DEXP3 algorithm in Saha et al. (2021b), which deals with the adversarial multi-armed dueling bandit. Algorithm 2 maintains a distribution q t ( · ) over [ K ] , initialized as uniform distribution (Line 2). At every round t , two items are chosen following q t independently.

Then Line 6 calculates the one-sample unbiased estimate ̂ θ t of the true underlying parameter θ ∗ t . Line 7 further calculates the unbiased estimate of the (shifted) Borda score. Note that the true Borda score at round t satisfies B t ( i ) = 1 2 + ⟨ 1 K ∑ j ∈ [ K ] ϕ i,j , θ ∗ t ⟩ . ̂ B t instead only estimates the second term of the Borda score. This is a choice to simplify the proof. The cumulative estimated score ∑ t l =1 ̂ B l ( i ) can be seen as the estimated cumulative reward of item i at round t . In Line 8, q t +1 is defined by the classic exponential weight update, along with a uniform exploration policy controlled by γ .

6.2 Upper Bounds

Algorithm 2 can also be configured to tightly match the worst-case lower bound:

Theorem 9. Suppose Assumption 1 holds. If we set η = (log K ) 2 / 3 d -1 / 3 T -2 / 3 and γ = √ ηd/λ 0 = (log K ) 1 / 3 d 1 / 3 T -1 / 3 λ -1 / 2 0 , then the expected regret is upper-bounded by

Note that the lower bound construction is for the linear model and has K = O (2 d ) , thus exactly 297 matching the upper bound. 298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

7 Experiments

This section compares the proposed algorithm BETC-GLM with existing ones that are capable of minimizing Borda regret. We use random responses (generated from fixed preferential matrices) to interact with all tested algorithms. Each algorithm is run for 50 times over a time horizon of T = 10 6 . We report both the mean and the standard deviation of the cumulative Borda regret and supply some analysis. The following list summarizes all methods we studies in this section, a more complete description of the methods and parameters are available in Appendix E: BETC-GLM(-MATCH): Algorithm 1 proposed in this paper with different parameters. UCB-BORDA: The UCB algorithm (Auer et al.,

0

25000

50000

75000

100000

T

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Regret(T)

×

10

4

UCB-Borda

DEXP3

ETC-Borda

BETC-GLM

BETC-GLM-Match

BEXP3

(a) Generated Hard Case

0

25000

50000

75000

100000

T

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

Regret(T)

×

10

4

UCB-Borda

DEXP3

ETC-Borda

BETC-GLM

BETC-GLM-Match

BEXP3

(b) EventTime

2002) using Borda reduction . DEXP3: Dueling-Exp3 developed by Saha et al. (2021a). ETCBORDA: A simple explore-then-commit algorithm that does not take any contextual information into account. BEXP3: The proposed method for adversarial Borda bandits displayed in Algorithm 2.

Generated Hard Case We first test the algorithms on the hard instances constructed in Section 4. We generate θ ∗ randomly from {-∆ , +∆ } d with ∆ = 1 4 d so that the comparison probabilities p θ ∗ i,j ∈ [0 , 1] for all i, j ∈ [ K ] . We pick the dimension d = 6 and the number of arms is therefore K = 2 d +1 = 128 . Note the dual usage of d in our construction and the model setup in Section 3.1. We refer readers to Remark 11 in Appendix B for more details.

As depicted in Figure 2a, the proposed algorithms (BETC-GLM, BEXP3) outperform the baseline algorithms in terms of cumulative regret when reaching the end of time horizon T . For UCB-BORDA, since it is not tailored for the dueling regret definition, it suffers from a linear regret as its second arm is always sampled uniformly at random, leading to a constant regret per round. DEXP3 and ETC-BORDA are two algorithms designed for K -armed dueling bandits. Both are unable to utilize contextual information and thus demand more exploration. As expected, their regrets are higher than BETC-GLM or BEXP3.

Real-world Dataset To showcase the performance of the algorithms in a real-world setting, we use EventTime dataset (Zhang et al., 2016). In this dataset, K = 100 historical events are compared in a pairwise fashion by crowd-sourced workers. We first calculate the empirical preference probabilities ˜ p i,j from the collected responses, and construct a generalized linear model based on the empirical preference probabilities. The algorithms are tested under this generalized linear model. Due to space limitations, more details are deferred to Appendix F.

As depicted in Figure 2b, the proposed algorithm BETC-GLM outperforms the baseline algorithms in terms of cumulative regret when reaching the end of time horizon T . The other proposed algorithm BEXP3 performs equally well even when misspecified (the algorithm is designed for linear setting, while the comparison probability follows a logistic model).

8 Conclusion and Future Work

In this paper, we introduced Borda regret into the generalized linear dueling bandits setting, along with an explore-then-commit type algorithm BETC-GLM and an EXP3 type algorithm BEXP3. The algorithms can achieve a nearly optimal regret upper bound, which we corroborate with a matching lower bound. The theoretical performance of the algorithms is verified empirically. It demonstrates superior performance compared to other baseline methods.

For future works, due to the fact that our exploration scheme guarantees an accurate estimate in all directions, our work can be extended to solve the top-k recovery or ranking problem, as long as a proper notion of regret can be identified.

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

References

ABBASI-YADKORI, Y., PÁL, D. and SZEPESVARI, C. (2011). Improved algorithms for linear stochastic bandits. In NIPS .

AUER, P., CESA-BIANCHI, N. and FISCHER, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning 47 235-256.

BALSUBRAMANI, A., KARNIN, Z., SCHAPIRE, R. E. and ZOGHI, M. (2016). Instance-dependent regret bounds for dueling bandits. In Conference on Learning Theory . PMLR.

BENGS, V., BUSA-FEKETE, R., EL MESAOUDI-PAUL, A. and HÜLLERMEIER, E. (2021). Preference-based online learning with dueling bandits: A survey. Journal of Machine Learning Research 22 7-1.

BUSA-FEKETE, R., HÜLLERMEIER, E. and MESAOUDI-PAUL, A. E. (2018). Preference-based online learning with dueling bandits: A survey. ArXiv abs/1807.11398 .

CHEN, X., BENNETT, P. N., COLLINS-THOMPSON, K. and HORVITZ, E. (2013). Pairwise ranking aggregation in a crowdsourced setting. In Proceedings of the sixth ACM international conference on Web search and data mining .

DANI, V., HAYES, T. P. and KAKADE, S. M. (2008). Stochastic linear optimization under bandit feedback. In Annual Conference Computational Learning Theory .

DUDÍK, M., HOFMANN, K., SCHAPIRE, R. E., SLIVKINS, A. and ZOGHI, M. (2015). Contextual dueling bandits. ArXiv abs/1502.06362 .

EVEN-DAR, E., MANNOR, S. and MANSOUR, Y. (2002). Pac bounds for multi-armed bandit and markov decision processes. In Annual Conference Computational Learning Theory .

FALAHATGAR, M., HAO, Y., ORLITSKY, A., PICHAPATI, V. and RAVINDRAKUMAR, V. (2017a). Maxing and ranking with few assumptions. Advances in Neural Information Processing Systems 30 .

FALAHATGAR, M., JAIN, A., ORLITSKY, A., PICHAPATI, V. and RAVINDRAKUMAR, V. (2018). The limits of maxing, ranking, and preference learning. In International conference on machine learning . PMLR.

FALAHATGAR, M., ORLITSKY, A., PICHAPATI, V. and SURESH, A. T. (2017b). Maximum selection and ranking under noisy comparisons. In International Conference on Machine Learning . PMLR.

FAURY, L., ABEILLE, M., CALAUZÈNES, C. and FERCOQ, O. (2020). Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning . PMLR.

FILIPPI, S., CAPPE, O., GARIVIER, A. and SZEPESVÁRI, C. (2010). Parametric bandits: The generalized linear case. Advances in Neural Information Processing Systems 23 .

HECKEL, R., SIMCHOWITZ, M., RAMCHANDRAN, K. and WAINWRIGHT, M. (2018). Approximate ranking from pairwise comparisons. In International Conference on Artificial Intelligence and Statistics . PMLR.

JAMIESON, K., KATARIYA, S., DESHPANDE, A. and NOWAK, R. (2015). Sparse dueling bandits. In Artificial Intelligence and Statistics . PMLR.

JUN, K.-S., BHARGAVA, A., NOWAK, R. and WILLETT, R. (2017). Scalable generalized linear bandits: Online computation and hashing. Advances in Neural Information Processing Systems 30 .

KOMIYAMA, J., HONDA, J. and NAKAGAWA, H. (2016). Copeland dueling bandit problem: Regret lower bound, optimal algorithm, and computationally efficient algorithm. In International Conference on Machine Learning . PMLR.

KULESHOV, V. and PRECUP, D. (2014). Algorithms for multi-armed bandit problems. arXiv preprint arXiv:1402.6028 .

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

LAI, T. L., ROBBINS, H. ET AL. (1985). Asymptotically efficient adaptive allocation rules. Advances in applied mathematics 6 4-22.

LATTIMORE, T. and SZEPESVÁRI, C. (2020). Bandit Algorithms . Cambridge University Press.

LI, L., LU, Y. and ZHOU, D. (2017). Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning . PMLR.

LIU, C., JIN, T., HOI, S. C. H., ZHAO, P. and SUN, J. (2017). Collaborative topic regression for online recommender systems: an online and bayesian approach. Machine Learning 106 651-670.

LOU, H., JIN, T., WU, Y., XU, P., GU, Q. and FARNOUD, F. (2022). Active ranking without strong stochastic transitivity. Advances in neural information processing systems .

MINKA, T. P., CLEVEN, R. and ZAYKOV, Y. (2018). Trueskill 2: An improved bayesian skill rating system.

RAMAMOHAN, S., RAJKUMAR, A. and AGARWAL, S. (2016). Dueling bandits: Beyond condorcet winners to general tournament solutions. In NIPS .

REN, W., LIU, J. K. and SHROFF, N. (2019). On sample complexity upper and lower bounds for exact ranking from noisy comparisons. Advances in Neural Information Processing Systems 32 .

RUSMEVICHIENTONG, P. and TSITSIKLIS, J. N. (2010). Linearly parameterized bandits. Mathematics of Operations Research 35 395-411.

SAHA, A. (2021). Optimal algorithms for stochastic contextual preference bandits. Advances in Neural Information Processing Systems 34 30050-30062.

SAHA, A., KOREN, T. and MANSOUR, Y. (2021a). Adversarial dueling bandits. ArXiv abs/2010.14563 .

SAHA, A., KOREN, T. and MANSOUR, Y. (2021b). Adversarial dueling bandits. In International Conference on Machine Learning . PMLR.

SUI, Y. and BURDICK, J. (2014). Clinical online recommendation with subgroup rank feedback. In Proceedings of the 8th ACM conference on recommender systems .

URVOY, T., CLÉROT, F., FÉRAUD, R. and NAAMANE, S. (2013). Generic exploration and k-armed voting bandits. In ICML .

VASILE, F., SMIRNOVA, E. and CONNEAU, A. (2016). Meta-prod2vec: Product embeddings using side-information for recommendation. In Proceedings of the 10th ACM conference on recommender systems .

WANG, J., HUANG, P., ZHAO, H., ZHANG, Z., ZHAO, B. and LEE (2018). Billion-scale commodity embedding for e-commerce recommendation in alibaba. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining .

WU, H. and LIU, X. (2016). Double thompson sampling for dueling bandits. ArXiv abs/1604.07101 .

WU, Y., JIN, T., LOU, H., XU, P., FARNOUD, F. and GU, Q. (2022). Adaptive sampling for heterogeneous rank aggregation from noisy pairwise comparisons. In International Conference on Artificial Intelligence and Statistics . PMLR.

YUE, Y., BRODER, J., KLEINBERG, R. D. and JOACHIMS, T. (2012). The k-armed dueling bandits problem. J. Comput. Syst. Sci. 78 1538-1556.

YUE, Y. and JOACHIMS, T. (2009). Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th Annual International Conference on Machine Learning .

YUE, Y. and JOACHIMS, T. (2011). Beat the mean bandit. In International Conference on Machine Learning .

438

439

440

441

442

443

444

445

446

447

ZHANG, X., LI, G. and FENG, J. (2016). Crowdsourced top-k algorithms: An experimental evaluation. Proc. VLDB Endow. 9 612-623.

ZHU, J., AHMED, A. and XING, E. P. (2012). Medlda: maximum margin supervised topic models. J. Mach. Learn. Res. 13 2237-2278.

ZOGHI, M., KARNIN, Z. S., WHITESON, S. and DE RIJKE, M. (2015). Copeland dueling bandits. In NIPS .

ZOGHI, M., WHITESON, S., MUNOS, R. and DE RIJKE, M. (2014a). Relative upper confidence bound for the k-armed dueling bandit problem. ArXiv abs/1312.3393 .

ZOGHI, M., WHITESON, S., MUNOS, R. and RIJKE, M. (2014b). Relative upper confidence bound for the k-armed dueling bandit problem. In International conference on machine learning . PMLR.

Peer reviews:
comments: SUMMARY:
The authors consider the linear dueling bandit problem, where the regret is measured in terms of Borda regret. The Borda regret function was adopted as that used in Saha et al. 2021a. Different from conventional bandit settings, there is a mismatch between the reward function and the regret function definition in the sense that the expected reward of pulling a pair of arms is p_i,j, while the regret is defined based on the function B(i)+B(j), where B(i) is the Borda score of i-th arm. The authors provide a lower bound on the regret by the construction of hard cases and a matching algorithm based on explore-then-commit. The adversarial setting is also considered, where the EXP3 style algorithm is proposed and analyzed, and shown to match the upper bound.

STRENGTHS:
+ For both settings, the proposed algorithms are shown to match the upper bounds.
+ The construction of hard cases is interesting.

WEAKNESSES:
- The experiment results are not very convincing.
- The algorithms are relatively straightforward extensions of the existing approaches
- The constructed hard case is also a relatively straightforward extension of that in Saha et al.

QUESTIONS:
- The setup for experiments is confusing. The two algorithms are for single context and adversarial settings, respectively. It is not clear why they can be used in the same setting for evaluation.

line 8 of algorithm 1: "s" should be "t"

LIMITATIONS:
not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of minimising Borda regret for dueling bandits in the generalised linear setting where each pair of arms has a context vector associated with it.\n\nThe paper considers both the stochastic setting where the parameter $\\theta^*$ used for generating rewards is fixed and the adversarial setting where it is time dependent.\n\n**[Lower bound]** The authors show a hard problem instance for which any algorithm will incur a regret of $\\Omega(d^{2/3}T^{2/3})$ in the stochastic setting, $d$ being the dimension of the context vectors.\n\n**[BETC-GLM for stochastic setting]** The authors propose an explore-then-commit style algorithm that first pulls arm pairs based on a G-optimal design for some rounds and then exploits by selecting the arm $\\hat{i}$ with the highest MLE Borda score and pulling $(\\hat{i}, \\hat{i})$. This algorithm incurs $\\tilde{O}(\\kappa^{-1} d^{2/3} T^{2/3})$ regret, where $\\kappa$ is a problem dependent parameter (which is constant for linear bandits).\n\n**[BEXP3]** The authors then propose an EXP style algorithm for the adversarial case. The algorithm is restricted to the linear setting. At each step, it estimates $\\hat{\\theta}_t$ using the reward obtained in that step, and uses it to estimate the Borda scores of all arms at that step. These scores are used to compute a distribution over arm pairs in the same spirit at EXP. This algorithm incurs a regret of $O((d \\log K)^{1/3} T^{2/3})$, which matches the lower bound when $K = O(2^d)$.\n\nExperiments on real and synthetic data corroborate the theoretical findings.\n\nSTRENGTHS:\nOriginality - The Borda regret minimisation problem for generalised linear dueling bandits is new. The authors seem to have adequately cited the related work.\n\nQuality - I have not checked the details in the appendix but the arguments presented in the main paper are inspired from well-known techniques, and appear sound to me. Barring a few issues listed in the weakness section, I believe that the claims are well supported, both theoretically and empirically via experiments on real and synthetic data.\n\nClarity - The paper is very well written and easy to understand. In particular, I appreciate the explanation for ``Borda reduction'' and why it is not sufficient in Section 3.1.\n\nSignificance - The results fill an important research gap.\n\nWEAKNESSES:\nOriginality - The algorithms and their analysis have limited novelty. While I understand why Borda reduction does not trivially work, it would be helpful to have a summary of the technical challenges encountered in analysing the algorithms, while highlighting the new ideas that were employed.\n\nQuality - BEXP3 assumes a linear model for $p_{i,j}^t$ and not a generalised linear model. I think this should be clarified in the abstract. It would also be useful to have a discussion on what makes having a link function here hard.\n\nClarity - Just two minor comments, \n1. Use \\citep instead of \\citet wherever appropriate (e.g., L19-20)\n2. Explaining the utility of using a G-optimal design in Algorithm 1 (to have a uniformly good estimate of $\\theta$) will improve the readability of the paper.\n\nSignificance - The authors note in their conclusion that their exploration scheme guarantees accurate estimation in all directions, thereby paving way for extensions like top-k recovery and ranking problems. However, the exploration scheme is the G-optimal design, which is not a new contribution. Outside of this, the work seems to have limited impact.\n\nQUESTIONS:\nPlease address the points under originality and significance in the weakness section.\n\nLIMITATIONS:\nI suggest adding some discussion addressing my concern under \"Quality\" in the weakness section as a limitation of the analysis for BEXP3 (if applicable, I may be wrong in which case please correct me).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses Borda regret minimization in a contextual dueling bandits scenario, where the context is given in a generalized linear form.\nIt provides a worst-case lower bound for the stochastic and adversarial learning scenario. The authors develop for both scenarios algorithmic solutions whose asymptotic Borda regret matches these lower bounds (up to logarithmic terms). These solutions are shown to empirically outperform current state-of-the art solutions in experiments.\n\nSTRENGTHS:\nThe paper tackles an interesting problem. It contributes a novel lower bound for the particular learning scenario and develops algorithmic solutions, which (a) come with log-optimal regret upper bounds and (b) outperform current methods on both synthetic and real-world data.\nIn my opinion, this paper is well-written, the notation is convenient and the algorithms and proofs seem to be presented in a reader-friendly way.\n\nWEAKNESSES:\nI did not find any weaknesses while skimming the paper and the proofs.\n\nWhile reading, I've collected some the following typos/minor suggestions:\n- 170f.: $\\lambda_{min}$ has not formally been introduced.\n- 216: Refer here to the appendix for the proof of Thm. 4. Or did you mention before that proofs are to be found in the appendix?\n- 240f.: Here, you call the G-optimal design $\\pi^\\ast$. For consistency, denote it also like this in Alg. 1 and the further discussion?\n- 297: Refer to Thm.4 for convenience?\n- 309: \"we study\"\n- 331: [the] EventTime dataset\n- 338: for [the/a] linear setting\n- 457: be satisfied\n- 522f.: \\P_{\\theta,\\A} has not been defined\n- 522f.: C=10?\n- 529f.: Should averaging be done over $\\theta \\in \\{-\\Delta,\\Delta\\}$? Also, in 532ff. etc.\n- 532f.: Why is \"sign\" in bold?\n- 552f.: Regarding your identity of $V_\\tau$ here, you seem to have switched from column- to rows for $\\theta_{i,j}$ in comparison to l. 171f.\n- 568f.: For readability, formally introduce $\\succeq$\n- 582: $|\\mathrm{supp}(\\pi)|$\n\nQUESTIONS:\nDoes the hard instance in Remark 11 fulfill all of assumptions 1-3? (If trivial, at least mention it briefly for convenience.) If not, could you provide another hard instance fulfilling these, which leads to a similar bound?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn dueling bandits, each pair of arms corresponds to some unknown probability $p_{i,j}$ where $p_{i,j}$ is the probability arm i is ranked higher than $j$. The learner sequentially chooses pairs of arms and receives a noisy result as to the ordering of the pair, i.e. Bernoulli$(p_{i,j})$. This paper considers dueling bandits under a generalised linear model where each pair of arms $i,j$ corresponds to a known context vector. The probability that $i$ is ranked higher $j$ is then the product of this context vector with some unknown vector $\\theta^*$, passed through a link function. The goal of the learner is to pull arms with high ranking as much as possible, the regret on a single pair of arms is given as the average Borda score between the arms, and the learner aims to minimise cumulative regret. This setting has been considered previously, however, under the assumption that a hidden coherent ordering is forced on the arms. The key contribution of this paper is to relax this assumption and have essentially no constraint on the $p_{i,j}$s. In addition to the above setting, the authors consider a adversarial setting where the unknown vector $\\theta^*$ is allowed to change round by round. Adversarial dueling bandits has been studied in the literature, the main contribution of this section is to extend this analysis to the contextual setting.\n\nSTRENGTHS:\nThe paper is well written and easy to read. Sufficient treatment is given to previous works and care is taken to illustrate how the results of this paper are novel in comparison. In section 4 the class of hard instances is clearly constructed and the following discussion gives good intuition as to why explore then commit algorithms can work in this setting, which is in itself an interesting phenomenon. The results give a complete treatment of the setting, with matching upper and lower bounds, up to log terms. In their experiments the authors consider a variety of benchmarks, with application to synthetic and real world data.\n\nWEAKNESSES:\nI do not see why $\\delta$ is given as input to the algorithm, it is not taken as a parameter but rather passed to the exploration phase $\\tau$. \n\nThe discussion in section A.1 of the appendix is vital to understanding the novelty of this work in comparison to Saha 2021, it is a shame that this section is not present in the main text.\n\nQUESTIONS:\nHave the authors considered relaxed assumptions, that do not require a coherent ordering of the arms, but ensure that explore then commit algorithms cannot be optimal? For instance, a arm cannot be ranked higher than another arm whose Borda score is sufficiently higher than its own, according to some tolerance.\n\nLIMITATIONS:\nFor the BETC-GLM algorithm, the authors acknowledge the potential limitation of not having access to the exact G optimal design. They suggest a well known sub routine to estimate the G optimal design and describe the additional error term that would incurred by this. The authors discuss and provide compelling arguments as to whether the assumptions are reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}