Title: Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback
Venue: NeurIPS 2024
Accepted: no
Keywords: Dueling bandits

bandits

preference feedback

adversarially robust

weighted MLE
Abstract: Learning from human feedback plays an important role in aligning generative models, such as large language models (LLM). However, the effectiveness of this approach can be influenced by adversaries, who may intentionally provide misleading preferences to manipulate the output in an undesirable or harmful direction.
To tackle this challenge, we study a specific model within this problem domain--contextual dueling bandits with adversarial feedback, where the true preference label can be flipped by an adversary. We propose an algorithm namely robust contextual dueling bandit (\algo), which is based on uncertainty-weighted maximum likelihood estimation.  Our algorithm achieves an $\tilde O(d\sqrt{T}+dC)$ regret bound, where $T$ is the number of rounds, $d$ is the dimension of the context, and $  0 \le C \le T$ is the total number of adversarial feedback. We also prove a lower bound to show that our regret bound is nearly optimal, both in scenarios with and without ($C=0$) adversarial feedback.
Additionally, we conduct experiments to evaluate our proposed algorithm against various types of adversarial feedback. Experimental results demonstrate its superiority over the state-of-the-art dueling bandit algorithms in the presence of adversarial feedback.

Abstract

Learning from human feedback plays an important role in aligning generative models, such as large language models (LLM). However, the effectiveness of this approach can be influenced by adversaries, who may intentionally provide misleading preferences to manipulate the output in an undesirable or harmful direction. To tackle this challenge, we study a specific model within this problem domain-contextual dueling bandits with adversarial feedback, where the true preference label can be flipped by an adversary. We propose an algorithm namely robust contextual dueling bandits ( RCDB ), which is based on uncertainty-weighted maximum likelihood estimation. Our algorithm achieves an ˜ O ( d √ T + dC ) regret bound, where T is the number of rounds, d is the dimension of the context, and 0 ≤ C ≤ T is the total number of adversarial feedback. We also prove a lower bound to show that our regret bound is nearly optimal, both in scenarios with and without ( C = 0 ) adversarial feedback. Additionally, we conduct experiments to Experimental results demonstrate its superiority over the state-of-the-art dueling evaluate our proposed algorithm against various types of adversarial feedback. bandit algorithms in the presence of adversarial feedback.

1 Introduction

Acquiring an appropriate reward proves challenging in numerous real-world applications, often 18 necessitating intricate instrumentation (Zhu et al., 2020) and time-consuming calibration (Yu et al., 19 2020) to achieve satisfactory levels of sample efficiency. For instance, in training large language 20 models (LLM) using reinforcement learning from human feedback (RLHF), the diverse values and 21 perspectives of humans can lead to uncalibrated and noisy rewards (Ouyang et al., 2022). In contrast, 22 preference-based data, which involves comparing or ranking various actions, is a more straightforward 23 method for capturing human judgments and decisions. In this context, the dueling bandit model 24 (Yue et al., 2012) provides a problem framework that focuses on optimal decision-making through 25 pairwise comparisons, rather than relying on the absolute reward for each action. 26 However, human feedback may not always be reliable. In real-world applications, human feedback 27 is particularly vulnerable to manipulation through preference label flip. Adversarial feedback can 28 significantly increase the risk of misleading a large language model (LLM) into erroneously prioritiz29 ing harmful content, under the false belief that it reflects human preference. Despite the significant 30 influence of adversarial feedback, there is limited existing research on the impact of adversarial 31 feedback specifically within the context of dueling bandits. A notable exception is Agarwal et al. 32 (2021), which studies dueling bandits when an adversary can flip some of the preference labels 33 received by the learner. They proposed an algorithm that is agnostic to the amount of adversarial 34 feedback introduced by the adversary. However, their setting has the following two limitations. 35 First, their study was confined to a finite-armed setting, which renders their results less applicable 36 to modern applications such as RLHF. Second, their adversarial feedback is defined on the whole 37 comparison matrix. In each round, the adversary observes the outcomes of all pairwise comparisons 38

and then decides to corrupt some of the pairs before the agent selects the actions. This assumption 39

does not align well with the real-world scenario, where the adversary often flips the preference label 40 based on the information of the selected actions. 41

In this paper, to address the above challenge, we aim to develop contextual dueling bandit algorithms 42 that are robust to adversarial feedback. This enables us to effectively tackle problems involving 43 a large number of actions while also taking advantage of contextual information. We specifically 44 consider a scenario where the adversary knows the selected action pair and the true preference of 45 their comparison. In this setting, the adversary's only decision is whether to flip the preference label 46 or not. We highlight our contributions as follows: 47

48

49

50

51

52

We propose a new algorithm called robust contextual dueling bandits ( RCDB ), which integrates uncertainty-dependent weights into the Maximum Likelihood Estimator (MLE). Intuitively, our choice of weight is designed to induce a higher degree of skepticism about potentially 'untrustworthy' feedback. The agent is encouraged to focus more on feedback that is more likely to be genuine, effectively diminishing the impact of any adversarial feedback.

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

We analyze the regret of our algorithm under at most C number of adversarial feedback. Our result consists of two terms: a C -independent term ˜ O ( d √ T ) , which matches the lower bound established in Bengs et al. (2022) for uncorrupted linear contextual dueling bandits, and a C -dependent term ˜ O ( dC ) . Furthermore, we establish a lower bound for dueling bandits with adversarial feedback, demonstrating the optimality of our adversarial term. Consequently, our algorithm for dueling bandits attains the optimal regret in both scenarios, with and without adversarial feedback.

We conduct extensive experiments to validate the effectiveness of our algorithm RCDB . To comprehensively assess RCDB 's robustness against adversarial feedback, we evaluate its performance under various types of adversarial feedback and compare the results with state-of-the-art dueling bandit algorithms. Experimental results demonstrate the superiority of our algorithm in the presence of adversarial feedback, which corroborate our theoretical analysis.

̸

̸

Notation. In this paper, we use plain letters such as x to denote scalars, lowercase bold letters such as x to denote vectors and uppercase bold letters such as X to denote matrices. For a vector x , ∥ x ∥ 2 denotes its ℓ 2 -norm. The weighted ℓ 2 -norm associated with a positive-definite matrix A is defined as ∥ x ∥ A = √ x ⊤ Ax . For two symmetric matrices A and B , we use A ⪰ B to denote A -B is positive semidefinite. We use 1 to denote the indicator function and 0 to denote the zero vector. For two actions a , b , we use a ≻ b to denote a is more preferable to b . For a postive integer N , we use [ N ] to denote { 1 , 2 , . . . , N } . We use standard asymptotic notations including O ( · ) , Ω( · ) , Θ( · ) , and ˜ O ( · ) , ˜ Ω( · ) , ˜ Θ( · ) will hide logarithmic factors.

2 Related Work

Bandits with Adversarial Reward. The multi-armed bandit problem, involving an agent making sequential decisions among multiple arms, has been studied with both stochastic rewards (Lai et al., 1985; Lai, 1987; Auer, 2002; Auer et al., 2002a; Kalyanakrishnan et al., 2012; Lattimore and Szepesvári, 2020; Agrawal and Goyal, 2012), and adversarial rewards (Auer et al., 2002b; Bubeck et al., 2012). Moreover, a line of works focuses on designing algorithms that can achieve near-optimal regret bounds for both stochastic bandits and adversarial bandits simultaneously (Bubeck and Slivkins, 2012; Seldin and Slivkins, 2014; Auer and Chiang, 2016; Seldin and Lugosi, 2017; Zimmert and Seldin, 2019; Lee et al., 2021), which is known as 'the best of both worlds' guarantee. Distinct from

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

fully stochastic and fully adversarial models, Lykouris et al. (2018) studied a setting, where only a portion of the rewards is subject to corruption. They proposed an algorithm with a regret dependent on the corruption level C , defined as the cumulative sum of the corruption magnitudes in each round. Their result is C times worse than the regret without corruption. Gupta et al. (2019) improved the result by providing a regret guarantee comprising two terms, a corruption-independent term that matches the regret lower bound without corruption, and a corruption-dependent term that is linear in C . In addition, Gupta et al. (2019) proved a lower bound demonstrating the optimality of the linear dependency on C .

Contextual Bandits with Corruption. Li et al. (2019) studied stochastic linear bandits with corruption and presented an instance-dependent regret bound linearly dependent on the corruption level C . Bogunovic et al. (2021) studied the same problem and proposed an algorithm with nearoptimal regret in the non-corrupted case. Lee et al. (2021) studied this problem in a different setting, where the adversarial corruptions are generated through the inner product of a corrupted vector and the context vector. For linear contextual bandits, Bogunovic et al. (2021) proved that under an additional context diversity assumption, the regret of a simple greedy algorithm is nearly optimal with an additive corruption term. Zhao et al. (2021) and Ding et al. (2022) extended the OFUL algorithm (Abbasi-Yadkori et al., 2011) and proved a regret with a corruption term polynomially dependent on the total number of rounds T . He et al. (2022) proposed an algorithm for known corruption level C to remove the polynomial dependency on T in the corruption term, which only has a linear dependency on C . They also proved a lower bound showing the optimality of linear dependency on C for linear contextual bandits with a known corruption level. Additionally, He et al. (2022) extended the proposed algorithm to an unknown corruption level and provided a near-optimal performance guarantee that matches the lower bound. For more extensions, Kuroki et al. (2023) studied best-of-both-worlds algorithms for linear contextual bandits. Ye et al. (2023) proposed a corruption robust algorithm for nonlinear contextual bandits.

Dueling Bandits and Logistic Bandits. The dueling bandit model was first proposed in Yue et al. (2012). Compared with bandits, the agent will select two arms and receive the preference feedback between the two arms from the environment. For general preference, there may not exist the 'best' arm that always wins in the pairwise comparison. Therefore, various alternative winners are considered, including Condorcet winner (Zoghi et al., 2014; Komiyama et al., 2015), Copeland winner (Zoghi et al., 2015; Wu and Liu, 2016; Komiyama et al., 2016), Borda winner (Jamieson et al., 2015; Falahatgar et al., 2017; Heckel et al., 2018; Saha et al., 2021; Wu et al., 2023) and von Neumann winner (Ramamohan et al., 2016; Dudík et al., 2015; Balsubramani et al., 2016), along with their corresponding performance metrics. To handle potentially large action space or context information, Saha (2021) studied a structured contextual dueling bandit setting. In this setting, each arm possesses an unknown intrinsic reward. The comparison is determined based on a logistic function of the relative rewards. In a similar setting, Bengs et al. (2022) studied contextual linear stochastic transitivity model with contextualized utilities. Di et al. (2023) proposed a layered algorithm with variance aware regret bound. Another line of works does not make the reward assumption. Instead, they assume the preference feedback can be represented by a function class. Saha and Krishnamurthy (2022) designed an algorithm that achieves the optimal regret for K -armed contextual dueling bandit problem. Sekhari et al. (2023) studied contextual dueling bandits in a more general setting and proposed an algorithm the provides guarantees for both regret and the number of queries. Another related area of research is the logistic bandits, where the agent selects one arm in each round and receives a Bernoulli reward. Faury et al. (2020) studied the dependency with respect to the degree of non-linearity of the logistic function κ . They proposed an algorithm with no dependency in κ . Abeille et al. (2021) further improved the dependency on κ and proved a problem dependent lower bound. Faury et al. (2022) proposed a computationally efficient algorithm with regret performance still matching the lower-bound proved in Abeille et al. (2021).

Dueling Bandits with Adversarial Feedback. A line of work has focused on dueling bandits with adversarial feedback or corruption. Gajane et al. (2015) studied a fully adversarial utility-based version of dueling bandits, which was proposed in Ailon et al. (2014). Saha et al. (2021) considered the Borda regret for adversarial dueling bandits without the assumption of utility. In a setting parallel to that in Lykouris et al. (2018); Gupta et al. (2019), Agarwal et al. (2021) studied K -armed dueling bandits in a scenario where an adversary has the capability to corrupt part of the feedback received by the learner. They designed an algorithm whose regret comprises two terms: one that is optimal in uncorrupted scenarios, and another that is linearly dependent on the total times of adversarial feedback C . Later on, Saha and Gaillard (2022) achieved 'best-of-both world' result for noncontextual dueling bandits and improved the adversarial term of Agarwal et al. (2021) in the same

setting. For contextual dueling bandits, Wu et al. (2023) proposed an EXP3-type algorithm for the 140 adversarial linear setting using Borda regret. For a comparison of the most related works for robust 141 bandits and dueling bandits, please refer to Table 1. In this paper, we study the influence of adversarial 142 feedback within contextual dueling bandits, particularly in a setting where only a minority of the 143 feedback is adversarial. Compared to previous studies, most studies have focused on the multi-armed 144 dueling bandit framework without integrating context information. The notable exception is Wu et al. 145 (2023); however, this study does not provide guarantees regarding the dependency on the number of 146 adversarial feedback instances. 147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

3 Preliminaries

In this work, we study linear contextual dueling bandits with adversarial feedback. In each round t ∈ [ T ] , the agent observes the context information x t from a context set X and the corresponding action set A . Utilizing this context information, the agent selects two actions, a t and b t . Subsequently, the environment will generate a binary feedback (i.e., preference label) l t = 1 ( a t ≻ b t ) ∈ { 0 , 1 } indicating the preferable action. We assume the existence of a reward function r ∗ ( x, a ) dependent on the context information x and action a , and a monotonically increasing link function σ satisfying σ ( x ) + σ ( -x ) = 1 . The preference probability will be determined by the link function and the difference between the rewards of the selected arms, i.e.,

We assume that the reward function is linear with respect to some known feature map ϕ ( x, a ) . To be more specific, we make the following assumption:

Assumption 3.1. Let ϕ : X × A → R d be a known feature map, with ∥ ϕ ( x, a ) ∥ 2 ≤ 1 for any ( x, a ) ∈ X × A . We define the reward function r θ parameterized by θ ∈ R d , with r θ ( x, a ) = ⟨ θ , ϕ ( x, a ) ⟩ . Moreover, there exists θ ∗ satisfying r θ ∗ = r ∗ , with ∥ θ ∗ ∥ 2 ≤ B .

Similar assumptions have been made in the literature of dueling bandits (Saha, 2021; Bengs et al., 2022; Xiong et al., 2023). We also make an assumption on the derivative of the link function, which is common in the study of generalized linear models for bandits (Filippi et al., 2010).

Assumption 3.2. The link function σ is differentiable. Furthermore, its first-order derivative satisfies:

for some constant κ > 0 .

In our setting, however, the agent does not directly observe the true binary feedback. Instead, an adversary will see both the choice of the agent and the true feedback. Based on the information, the adversary can decide whether to corrupt the binary feedback or not. 1 We represent the adversary's decision in round t by an adversarial indicator c t , which takes values from the set { 0 , 1 } . If the adversary chooses not to corrupt the result, we have c t = 0 . Otherwise, we have c t = 1 , which means adversarial feedback in this round. As a result, the agent will observe a flipped preference label, i.e., the observation o t = 1 -l t . We define C as the total level of adversarial feedback, i.e.,

Remark 3.3. Adversarial corruption has been firstly studied in bandits (Lykouris et al., 2018), where in each round t , the agent selects an action a t and the environment generates a numerical reward r t ( a t ) . The adversary observes the reward and returns a corrupted reward ¯ r t . The corruption level C is defined by ∑ T t =1 | r t ( a t ) -¯ r t | ≤ C . Compared with the continuous perturbation of rewards in bandits, the adversary's label flipping attack method in our model is quite different. The cost of obtaining adversarial feedback is uniformly 1, unlike in bandits where the cost depends on the intensity of the perturbation. Additionally, adversarial feedback in our setting involves comparing two arms, whereas in bandits it pertains to the reward of a single arm. The only previous work that studied label-flipping is (Agarwal et al., 2021), where the adversary cannot observe the action selected by the agent. In contrast, our setting focuses on scenarios where this information is available to adversaries, which is common in many real-life applications.

As the context is changing, the optimal action is different in each round, denoted by a ∗ t = argmax a ∈A r ∗ ( x t , a ) . The goal of our algorithm is to minimize the cumulative gap between the rewards of both selected actions and the optimal action

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

This regret definition is the same as that in Saha (2021) and the average regret defined in Bengs et al. (2022). It is typically stronger than weak regret defined in Bengs et al. (2022), which only considers the reward gap of the better action.

4 Algorithm

In this section, we present our new algorithm RCDB , designed for learning contextual linear dueling bandits. The main algorithm is illustrated in Algorithm 1. At a high level, we incorporate uncertaintydependent weighting into the Maximum Likelihood Estimator (MLE) to counter adversarial feedback. Specifically, in each round t ∈ [ T ] , we construct the estimator of parameter θ by solving the following equation:

where we denote ϕ i = ϕ ( x i , a i ) -ϕ ( x i , b i ) for simplicity, w i is the uncertainty weight we are going to choose. To obtain an intuitive understanding of our weight, we consider any action-observation sequence ( x 1 , a 1 , b 1 , o 1 , x 2 , a 2 , b 2 , o 2 , . . . , x t , a t , b t , o t ) up to round t . For simplicity, we denote F t = σ ( x 1 , a 1 , b 1 , o 1 , x 2 , a 2 , b 2 , o 2 , . . . , x t , a t , b t ) as the filtration. Suppose the estimated parameter θ t is the solution to the unweighted version equation of (4.1), i.e.,

When we receive ϕ t = ϕ ( x t , a t ) -ϕ ( x t , b t ) , the probability of receiving l t = 1 can be estimated by σ ( ϕ ⊤ t θ t ) . We consider the conditional variance of the estimated probability σ ( ϕ ⊤ t θ t ) in round t , i.e.,Var [ σ ( ϕ ⊤ t θ t ) |F t ] , involving a posterior estimate of the prediction's variance. First, we have

Moreover, using the Taylor's expansion to (4.2), we have 205

Let Λ t = λκ I + ∑ t i =1 σ ′ ( ϕ ⊤ i θ ∗ ) ϕ i ϕ ⊤ i , we have 206

Therefore, the variance of the estimated preference probability can be approximated by 207

208

209

210

211

212

213

214

215

216

where the first inequality holds due to the Jensen's inequality and o 2 t = o t , and the last inequality holds due to E [ o t |F t ] ≤ 1 . Using σ ′ ( ϕ ⊤ t θ ∗ ) ≤ 1 , ϕ ⊤ t θ ∗ ≤ 1 , Λ t ≥ κ Σ t +1 ≥ κ Σ t , we can see that Var [ σ ( ϕ ⊤ t θ t ) |F t ] ≤ κ -1 ∥ ϕ t ∥ 2 Σ -1 t . Since higher variance leads to larger uncertainty, which harms the credibility of the data, it is natural to assign a smaller weight to the data with high uncertainty. Thus, we choose the weight to cancel out the uncertainty as follows

where α/ ∥ ϕ i ∥ Σ -1 i normalizes the variance of the estimated probability. To prevent excessively large weights, we apply truncation to this value. A similar weight has been used in He et al. (2022) for linear contextual bandits under corruption. Different from their setting where the weight is an estimate of the variance of the linear model, our weight is an estimate of a generalized linear model.

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

Furthermore, by selecting a proper threshold parameter, e.g., α = √ d/C , the weighted MLE shares the same confidence radius with that of the no-adversary scenario.

After constructing the estimator θ t from the weighted MLE, the sum of the estimated reward for each duel ( a, b ) can be calculated as ( ϕ ( x t , a ) + ϕ ( x t , b ) ) ⊤ θ t . To encourage the exploration of duel ( a, b ) with high uncertainty during the learning process, we introduce an exploration bonus with the following β ∥ ∥ ϕ ( x t , a ) -ϕ ( x t , b ) ∥ ∥ Σ -1 t , which follows a similar spirit to the bonus term in the context of linear bandit problems (Abbasi-Yadkori et al., 2011). However, the reward term and the bonus term exhibit different combinations of the feature maps ϕ ( x t , a ) and ϕ ( x t , b ) , which is the key difference between bandits and dueling bandits. The selection of action pairs ( a, b ) is subsequently determined by maximizing the estimated reward with the exploration bonus term, i.e.,

More discussion about the selection rule was discussed in Appendix A of Di et al. (2023).

Algorithm 1 Robust Contextual Dueling Bandit ( RCDB )

1: Require: α > 0 , Regularization parameter λ , confidence radius β .

2: for t = 1 , . . . , T do

3: Compute Σ t = λ I + ∑ t -1 i =1 w i ( ϕ ( x i , a i ) -ϕ ( x i , b i ) )( ϕ ( x i , a i ) -ϕ ( x i , b i ) ) ⊤ .

4: Calculate the MLE θ t by solving the following equation:

5: Observe the context vector x t .

6: Choose a t , b t = argmax a,b { ( ϕ ( x t , a ) + ϕ ( x t , b ) ) ⊤ θ t + β ∥ ∥ ϕ ( x t , a ) -ϕ ( x t , b ) ∥ ∥ Σ -1 t } .

7: The adversary sees the feedback l t = 1 ( a t ≻ b t ) and decides the indicator c t . Observe o t = l t when c t = 0 , otherwise observe o t = 1 -l t .

8: Set weight w t as (4.3).

9: end for

5 Main Results

5.1 Known Number of Adversarial Feedback

At the center of our algorithm design is the uncertainty-weighted MLE. When faced with adversarial feedback, the estimation error of the weighted MLE θ t can be characterized by the following lemma.

Lemma 5.1. If we set β = √ λB + ( αC + √ d log((1 + 2 T/λ ) /δ ) ) /κ , then with probability at least 1 -δ , for any t ∈ [ T ] , we have

Remark 5.2. If we set α = ( √ d + √ λB ) /C , then the bonus radius β has no direct dependency on the number of adversarial feedback C . This observation plays a key role in proving the adversarial term in the regret without polynomial dependence on the total number of rounds T .

With Lemma 5.1, we can present the following regret guarantee of our algorithm RCDB in the dueling bandit framework.

Theorem 5.3. Under Assumption 3.1 and 3.2, let 0 < δ < 1 , the total number of adversarial feedback be C . If we set the bonus radius to be √

then with probability at least 1 -δ , the regret in the first t rounds can be upper bounded by

Moreover, if we set α = ( √ d + √ λB ) /C , λ = 1 /B 2 , the regret upper bound can be simplified to 242 Regret ( T ) = O ( d √ T/κ + dC/κ ) .

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

Remark 5.4. Our regret bound consists of two terms. The first one is a C -independent term ˜ O ( d √ T ) , which matches the lower bound ˜ Ω( d √ T ) proved in Bengs et al. (2022). This indicates that our result is optimal in scenarios without adversarial feedback ( C = 0 ). Additionally, our result includes an additive term that is linearly dependent on the number of adversarial feedback C . When C = O ( √ T ) , the order of regret will be the same as the stochastic setting. It indicates the robustness of our algorithm to adversarial feedback. Additionally, the following theorem we present establishes a lower bound for this adversarial term, indicating that our dependency on the number of adversarial feedback C and the context dimension d is also optimal.

Theorem 5.5. For any dimension d , there exists an instance of dueling bandits with |A| = d , such that any algorithm with the knowledge of the number of adversarial feedback C must incur Ω( dC ) regret with probability at least 1/2.

Remark 5.6. The proof of Theorem 5.5 follows Bogunovic et al. (2021). In the constructed instances, only one action has reward 1, while others have 0. Compared with linear bandits, where the feedback is an exact reward, dueling bandits deal with the comparison between a pair of actions. A critical observation from our preference model, as formulated in (3.1), is that two actions with identical rewards result in a pair that is challenging to differentiate. The lower bound can be proved by corrupting every comparison into a random guess until the total times of adversarial feedback have been used up. For detailed proof, please refer to Section B.2. Our proved lower bound Ω( dC ) shows that our result is nearly optimal because of the linear dependency on C, d and only logarithmic dependency on the total number of rounds T .

5.2 Unknown Number of Adversarial Feedback

In our previous analysis, the selection of parameters depends on having prior knowledge of the total number of adversarial feedback C . In this subsection, we extend our previous result to address the challenge posed by an unknown number of adversarial feedback C . Our approach to tackle this uncertainty follows He et al. (2022), we introduce an adversarial tolerance threshold ¯ C for the adversary count. This threshold can be regarded as an optimistic estimator of the actual number of adversarial feedback C . Under this situation, the subsequent theorem provides an upper bound for regret of Algorithm 1 in the case of an unknown number of adversarial feedback C .

Theorem 5.7. Under Assumptions 3.1 and 3.2, if we set the the confidence radius as

√

with the pre-defined adversarial tolerance threshold ¯ C and α = ( d + λB ) / ¯ C , then with probability at least 1 -δ , the regret of Algorithm 1 can be upper bounded as following:

√

If the actual number of adversarial feedback C is smaller than the adversarial tolerance threshold ¯ C , then we have

If the actual number of adversarial feedback C is larger than the adversarial tolerance threshold ¯ C , then we have Regret ( T ) = O ( T ) .

Remark 5.8. The COBE framework (Wei et al., 2022) converts any algorithm with the known adversarial level to an algorithm in the unknown case. However, such a framework only works for weak adversaries and does not work in our strong adversary setting. In fact, He et al. (2022) proved that any algorithm cannot simultaneously achieve near-optimal regret when uncorrupted and maintain sublinear regret with corruption level C = Ω( √ T ) . Therefore, there exists a trade-off between robust adversarial defense and near-optimal algorithmic performance. Our algorithm achieves the same nearly optimal ˜ O ( d √ T ) regret as the no-adversary case even when C = Θ( √ T ) , which indicates that our results are optimal in the presence of an unknown number of adversarial feedback.

6 Experiments

6.1 Experiment Setup

Preference Model. We study the effect of adversarial feedback with the preference model determined by (3.1), where σ ( x ) = 1 / (1 + e -x ) . We randomly generate the underlying parameter in [ -0 . 5 , 0 . 5] d and normalize it to be a vector with ∥ θ ∗ ∥ 2 = 2 . Then, we set it to be the underlying parameter and construct the reward utilized in the preference model as r ∗ ( x, a ) = ⟨ θ ∗ , ϕ ( x, a ) ⟩ . √ √

We set the action set A = { -1 / d, 1 / d } d . For simplicity, we assume ϕ ( x, a ) = a . In our experiment, we set the dimension d = 5 , with the size of action set |A| = 2 d = 32 .

Adversarial Attack Methods. We study the performance of our algorithm using different adversar294 ial attack methods. We categorize the first two methods as 'weak' primarily because the adversary in 295 these scenarios does not utilize information about the agent's actions. In contrast, we classify the 296 latter two methods as 'strong' attacks. In these cases, the adversary leverages a broader scope of 297 information, including knowledge of the actions selected by the agent and the true preference model. 298 This enables it to devise more targeted adversarial methods. 299

300

301

'Greedy Attack": The adversary will flip the preference label for the first C rounds. After that, it will not corrupt the result anymore.

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

'Random Attack": In each round, the adversary will flip the preference label with the probability of 0 < p < 1 , until the times of adversarial feedback reach C .

'Adversarial Attack": The adversary can have access to the true preference model. It will only flip the preference label when it aligns with the preference model, i.e., the probability for the preference model to make that decision is larger than 0.5, until the times of adversarial feedback reach C .

'Misleading Attack": The adversary selects a suboptimal action. It will make sure this arm is always the winner in the comparison until the times of adversarial feedback reach C . In this way, it will mislead the agent to believe this action is the optimal one.

Experiment Setup. For each experiment instance, we simulate the interaction with the environment for T = 2000 rounds. In each round, the feedback for the action pair selected by the algorithm is generated according to the defined preference model. Subsequently, the adversary observes both the selected actions and their corresponding feedback and then engages in one of the previously described adversarial attack methods. We report the regret defined in (3.2) averaged across 10 random runs.

0

250

500

750

1000

1250

1500

1750

2000

Epoch

0

250

500

750

1000

1250

1500

1750

Regret

MaxInp

CoLSTIM

MaxPairUCB

RCDB

0

250

500

750

1000

1250

1500

1750

2000

Epoch

0

250

500

750

1000

1250

1500

1750

Regret

MaxInp

CoLSTIM

MaxPairUCB

RCDB

0

250

500

750

1000

1250

1500

1750

2000

Epoch

0

1000

2000

3000

4000

5000

Regret

MaxInp

CoLSTIM

MaxPairUCB

RCDB

0

250

500

750

1000

1250

1500

1750

2000

Epoch

0

2000

4000

6000

8000

Regret

Misleading Attack

MaxInp

CoLSTIM

MaxPairUCB

RCDB

(d) Misleading Attack

6.2 Performance Comparison

We first introduce the algorithms studied in this section.

MaxInP : Maximum Informative Pair by Saha (2021). It involves maintaining a standard MLE. With the estimated model, it then identifies a set of promising arms possible to beat the rest. The selection of arm pairs is then strategically designed to maximize the uncertainty in the difference between the two arms within this promising set, referred to as 'maximum informative'.

CoLSTIM : The method by Bengs et al. (2022). It involves maintaining a standard MLE for the estimated model. Based on this model, the first arm is selected as the one with the highest estimated reward, implying it is the most likely to prevail over competitors. The second arm is selected to be the first arm's toughest competitor, with an added uncertainty bonus.

MaxPairUCB : This algorithm was proposed in Di et al. (2023). It uses the regularized MLE to estimate the parameter θ ∗ . Then it selects the actions based on a symmetric action selection rule, i.e. the actions with the largest estimated reward plus some uncertainty bonus.

RCDB : Algorithm 1 proposed in this paper. The key difference from the other algorithms is the use of uncertainty weight in the calculation of MLE (4.4). The we use the same symmetric action selection rule as MaxPairUCB . Our experiment results show that the uncertainty weight is critical in the face of adversarial feedback.

Our results are demonstrated in Figure 1. In Figure 1(a) and Figure 1(b), we observe scenarios where the adversary is 'weak' due to the lack of access to information regarding the selected actions and the underlying preference model. Notably, in these situations, our algorithm RCDB outperforms all other

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

baseline algorithms, demonstrating its robustness. Among the other algorithms, CoLSTIM performs as the strongest competitor.

In Figure 1(c), the adversary employs a 'stronger' adversarial method. Due to the inherent randomness of the model, some labels may naturally be 'incorrect'. An adversary with knowledge of the selected actions and the preference model can strategically neglect these naturally incorrect labels and selectively flip the others. This method proves catastrophic for algorithms to learn the true model, as it results in the agent encountering only incorrect preference labels at the beginning. Our results indicate that this leads to significantly higher regret. However, it's noteworthy that our algorithm RCDB demonstrates considerable robustness.

In Figure 1(d), the adversary employs a strategy aimed at misleading algorithms into believing a suboptimal action is the best choice. The algorithm CoLSTIM appears to be the most susceptible to being cheated by this method. Despite the deployment of 'strong' adversarial methods, as shown in both Figure 1(c) and Figure 1(d), our algorithm, RCDB , consistently demonstrates exceptional robustness against these attacks. A significant advantage of RCDB lies in that our parameter is selected solely based on the number of adversarial feedback C , irrespective of the nature of the adversarial methods employed. This contrasts with other algorithms where parameter tuning must be specifically adapted for each distinct adversarial method.

25

50

75

100

125

150

175

200

Adversarial Feedback

500

1000

1500

2000

2500

3000

3500

Regret

Cumulative Regret versus Adversarial Feedback

MaxInp

CoLSTIM

MaxPairUCB

RCDB

6.3 Robustness to Different Numbers of Adversarial Feedback

In this section, we test the performance of algorithms with increasing times of adversarial feedback. Our results show a linear dependency on the number of adversarial feedback C , which is consistent with the theoretical results we have proved in Theorem 5.3 and 5.5. In comparison to other algorithms, RCDB demonstrates superior robustness against adversarial feedback, as evidenced by its notably smaller regret.

7 Conclusion

In this paper, we focus on the contextual dueling bandit problem from adversarial feedback. We introduce a novel algorithm, RCDB , which utilizes an uncertainty-weighted Maximum Likelihood Estimator (MLE) approach. This algorithm not only achieves optimal theoretical results in scenarios with and without adversarial feedback but also demonstrates superior performance with synthetic data. For future direction, we aim to extend our uncertainty-weighted method to encompass more general settings involving preference-based data. A particularly promising future direction of our research lies in addressing adversarial feedback within the process of aligning large language models using Reinforcement Learning from Human Feedback (RLHF).

Limitations. We assume that the reward is linear with respect to some known feature maps. Although this setting is common in the literature, we observe that some recent works on dueling bandits can deal with nonlinear rewards (Li et al., 2024). Therefore, it's possible to extend our results to a more general setting. Another assumption concerns the lower bound of the derivative of the link function. Notably, in the logistic bandit model, which shares similarities with our setting through Bernoulli variables, some work (Abeille et al., 2021; Faury et al., 2022) can improve the dependency of κ from 1 /κ to √ κ . A similar improvement might be achieved in our setting as well.

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

References

ABBASI-YADKORI, Y., PÁL, D. and SZEPESVÁRI, C. (2011). Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems .

ABEILLE, M., FAURY, L. and CALAUZÈNES, C. (2021). Instance-wise minimax-optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics . PMLR.

AGARWAL, A., AGARWAL, S. and PATIL, P. (2021). Stochastic dueling bandits with adversarial corruption. In Algorithmic Learning Theory . PMLR.

AGRAWAL, S. and GOYAL, N. (2012). Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory . JMLR Workshop and Conference Proceedings.

AILON, N., KARNIN, Z. and JOACHIMS, T. (2014). Reducing dueling bandits to cardinal bandits. In International Conference on Machine Learning . PMLR.

AUER, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research 3 397-422.

AUER, P., CESA-BIANCHI, N. and FISCHER, P. (2002a). Finite-time analysis of the multiarmed bandit problem. Machine Learning 47 235-256.

AUER, P., CESA-BIANCHI, N., FREUND, Y. and SCHAPIRE, R. E. (2002b). The nonstochastic multiarmed bandit problem. SIAM journal on computing 32 48-77.

AUER, P. and CHIANG, C.-K. (2016). An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits. In Conference on Learning Theory . PMLR.

BALSUBRAMANI, A., KARNIN, Z., SCHAPIRE, R. E. and ZOGHI, M. (2016). Instance-dependent regret bounds for dueling bandits. In Conference on Learning Theory . PMLR.

BENGS, V., SAHA, A. and HÜLLERMEIER, E. (2022). Stochastic contextual dueling bandits under linear stochastic transitivity models. In International Conference on Machine Learning . PMLR.

BOGUNOVIC, I., LOSALKA, A., KRAUSE, A. and SCARLETT, J. (2021). Stochastic linear bandits robust to adversarial attacks. In International Conference on Artificial Intelligence and Statistics . PMLR.

BUBECK, S., CESA-BIANCHI, N. ET AL. (2012). Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning 5 1-122.

BUBECK, S. and SLIVKINS, A. (2012). The best of both worlds: Stochastic and adversarial bandits. In Conference on Learning Theory . JMLR Workshop and Conference Proceedings.

CESA-BIANCHI, N. and LUGOSI, G. (2006). Prediction, learning, and games . Cambridge university press.

DI, Q., JIN, T., WU, Y., ZHAO, H., FARNOUD, F. and GU, Q. (2023). Variance-aware regret bounds for stochastic contextual dueling bandits. arXiv preprint arXiv:2310.00968 .

DING, Q., HSIEH, C.-J. and SHARPNACK, J. (2022). Robust stochastic linear contextual bandits under adversarial attacks. In International Conference on Artificial Intelligence and Statistics . PMLR.

DUDÍK, M., HOFMANN, K., SCHAPIRE, R. E., SLIVKINS, A. and ZOGHI, M. (2015). Contextual dueling bandits. In Conference on Learning Theory . PMLR.

FALAHATGAR, M., HAO, Y., ORLITSKY, A., PICHAPATI, V. and RAVINDRAKUMAR, V. (2017). Maxing and ranking with few assumptions. Advances in Neural Information Processing Systems 30 .

FAURY, L., ABEILLE, M., CALAUZÈNES, C. and FERCOQ, O. (2020). Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning . PMLR.

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

FAURY, L., ABEILLE, M., JUN, K.-S. and CALAUZÈNES, C. (2022). Jointly efficient and optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics . PMLR.

FILIPPI, S., CAPPE, O., GARIVIER, A. and SZEPESVÁRI, C. (2010). Parametric bandits: The generalized linear case. Advances in Neural Information Processing Systems 23 .

GAJANE, P., URVOY, T. and CLÉROT, F. (2015). A relative exponential weighing algorithm for adversarial utility-based dueling bandits. In International Conference on Machine Learning . PMLR.

GUPTA, A., KOREN, T. and TALWAR, K. (2019). Better algorithms for stochastic bandits with adversarial corruptions. In Conference on Learning Theory . PMLR.

HE, J., ZHOU, D., ZHANG, T. and GU, Q. (2022). Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. Advances in Neural Information Processing Systems 35 34614-34625.

HECKEL, R., SIMCHOWITZ, M., RAMCHANDRAN, K. and WAINWRIGHT, M. (2018). Approximate ranking from pairwise comparisons. In International Conference on Artificial Intelligence and Statistics . PMLR.

JAMIESON, K., KATARIYA, S., DESHPANDE, A. and NOWAK, R. (2015). Sparse dueling bandits. In Artificial Intelligence and Statistics . PMLR.

KALYANAKRISHNAN, S., TEWARI, A., AUER, P. and STONE, P. (2012). Pac subset selection in stochastic multi-armed bandits. In ICML , vol. 12.

KOMIYAMA, J., HONDA, J., KASHIMA, H. and NAKAGAWA, H. (2015). Regret lower bound and optimal algorithm in dueling bandit problem. In Conference on learning theory . PMLR.

KOMIYAMA, J., HONDA, J. and NAKAGAWA, H. (2016). Copeland dueling bandit problem: Regret lower bound, optimal algorithm, and computationally efficient algorithm. In International Conference on Machine Learning . PMLR.

KUROKI, Y., RUMI, A., TSUCHIYA, T., VITALE, F. and CESA-BIANCHI, N. (2023). Best-of-bothworlds algorithms for linear contextual bandits. arXiv preprint arXiv:2312.15433 .

LAI, T. L. (1987). Adaptive treatment allocation and the multi-armed bandit problem. The annals of statistics 1091-1114.

LAI, T. L., ROBBINS, H. ET AL. (1985). Asymptotically efficient adaptive allocation rules. Advances in applied mathematics 6 4-22.

LATTIMORE, T. and SZEPESVÁRI, C. (2020). Bandit Algorithms . Cambridge University Press.

LEE, C.-W., LUO, H., WEI, C.-Y., ZHANG, M. and ZHANG, X. (2021). Achieving near instanceoptimality and minimax-optimality in stochastic and adversarial linear bandits simultaneously. In International Conference on Machine Learning . PMLR.

LI, L., LU, Y. and ZHOU, D. (2017). Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning . PMLR.

LI, X., ZHAO, H. and GU, Q. (2024). Feel-good thompson sampling for contextual dueling bandits. arXiv preprint arXiv:2404.06013 .

LI, Y., LOU, E. Y. and SHAN, L. (2019). Stochastic linear optimization with adversarial corruption. arXiv preprint arXiv:1909.02109 .

LYKOURIS, T., MIRROKNI, V. and PAES LEME, R. (2018). Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing .

OUYANG, L., WU, J., JIANG, X., ALMEIDA, D., WAINWRIGHT, C., MISHKIN, P., ZHANG, C., AGARWAL, S., SLAMA, K., RAY, A. ET AL. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 2773027744.

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

RAMAMOHAN, S. Y., RAJKUMAR, A. and AGARWAL, S. (2016). Dueling bandits: Beyond condorcet winners to general tournament solutions. Advances in Neural Information Processing Systems 29 .

SAHA, A. (2021). Optimal algorithms for stochastic contextual preference bandits. Advances in Neural Information Processing Systems 34 30050-30062.

SAHA, A. and GAILLARD, P. (2022). Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences. In International Conference on Machine Learning . PMLR.

SAHA, A., KOREN, T. and MANSOUR, Y. (2021). Adversarial dueling bandits. In International Conference on Machine Learning . PMLR.

SAHA, A. and KRISHNAMURTHY, A. (2022). Efficient and optimal algorithms for contextual dueling bandits under realizability. In International Conference on Algorithmic Learning Theory . PMLR.

SEKHARI, A., SRIDHARAN, K., SUN, W. and WU, R. (2023). Contextual bandits and imitation learning via preference-based active queries. arXiv preprint arXiv:2307.12926 .

SELDIN, Y. and LUGOSI, G. (2017). An improved parametrization and analysis of the exp3++ algorithm for stochastic and adversarial bandits. In Conference on Learning Theory . PMLR.

SELDIN, Y. and SLIVKINS, A. (2014). One practical algorithm for both stochastic and adversarial bandits. In International Conference on Machine Learning . PMLR.

WEI, C.-Y., DANN, C. and ZIMMERT, J. (2022). A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory . PMLR.

WU, H. and LIU, X. (2016). Double thompson sampling for dueling bandits. Advances in neural information processing systems 29 .

WU, Y., JIN, T., LOU, H., FARNOUD, F. and GU, Q. (2023). Borda regret minimization for generalized linear dueling bandits. arXiv preprint arXiv:2303.08816 .

XIONG, W., DONG, H., YE, C., ZHONG, H., JIANG, N. and ZHANG, T. (2023). Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv:2312.11456 .

YE, C., XIONG, W., GU, Q. and ZHANG, T. (2023). Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In International Conference on Machine Learning . PMLR.

YU, T., QUILLEN, D., HE, Z., JULIAN, R., HAUSMAN, K., FINN, C. and LEVINE, S. (2020). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning . PMLR.

YUE, Y., BRODER, J., KLEINBERG, R. and JOACHIMS, T. (2012). The k-armed dueling bandits problem. Journal of Computer and System Sciences 78 1538-1556.

ZHAO, H., ZHOU, D. and GU, Q. (2021). Linear contextual bandits with adversarial corruptions. arXiv preprint arXiv:2110.12615 .

ZHU, H., YU, J., GUPTA, A., SHAH, D., HARTIKAINEN, K., SINGH, A., KUMAR, V. and LEVINE, S. (2020). The ingredients of real-world robotic reinforcement learning. arXiv preprint arXiv:2004.12570 .

ZIMMERT, J. and SELDIN, Y. (2019). An optimal algorithm for stochastic and adversarial bandits. In The 22nd International Conference on Artificial Intelligence and Statistics . PMLR.

ZOGHI, M., KARNIN, Z. S., WHITESON, S. and DE RIJKE, M. (2015). Copeland dueling bandits. Advances in neural information processing systems 28 .

ZOGHI, M., WHITESON, S., MUNOS, R. and RIJKE, M. (2014). Relative upper confidence bound for the k-armed dueling bandit problem. In International conference on machine learning . PMLR.

510

511

512

513

514

515

516

517

518

519

520

521

522

523

Broader Impact

This paper studies contextual dueling bandits with adversarial feedback. Our primary objective is to propel advancements in bandit theory by introducing a more robust algorithm backed by solid theoretical guarantees. The uncertainty-weighted approach we have developed for dueling bandits holds significant potential to address the issue of adversarial feedback in preference-based data, which could be instrumental in enhancing the robustness of generative models against adversarial attacks, thereby contributing positively to the societal impact and reliability of machine learning applications.

A Roadmap of the Proof

A.1 Uncertainty-weighted MLE with Adversarial Feedback

In this section, we offer an overview of the proof for Lemma 5.1. The general proof idea for the uncertainty-weighted MLE with adversarial feedback lies in decomposing the estimation error into three terms, a stochastic error term, an adversarial term, and an additional regularization term. Following the analysis of standard (weighted) MLE (Li et al., 2017), we introduce an auxiliary function:

It satisfies two conditions: First, for the true parameter value θ ∗ , G t ( θ ∗ ) has a simple expression, i.e., 524

Second, according to (4.4), we can get the value of function G t at the MLE θ t ,

where γ i = o i -σ ( ( ϕ ( x i , a i ) -ϕ ( x i , b i )) ⊤ θ ∗ ) . To connect the desired estimation error with the 526 function G t , we use the mean value theorem. This leads to an upper bound of the estimation error: 527

For term I 1 , we can decompose the summation in (A.1) based on the adversarial feedback c t , i.e., 528

where I 2 can be further decomposed as 529

where ϵ i = l i -σ ( ( ϕ ( x i , a i ) -ϕ ( x i , b i )) ⊤ θ ∗ ) . With our notation of adversarial feedback, when 530 c i = 0 , we have γ i = ϵ i . Therefore, we have | γ i -ϵ i | ≤ 1 and 531

The stochastic term can be upper bounded with the concentration inequality (Lemma D.2). Addition532 ally, by employing our specifically chosen weight, as (4.3), we can control the adversarial term, with 533 w i ∥ ϕ ( x i , a i ) -ϕ ( x i , b i ) ∥ Σ -1 t ≤ α . Therefore, the adversarial term can be bounded by αC/κ . 534

525

A.2 Regret Upper Bound 535

With a similar discussion of the symmetric arm selection rule to Di et al. (2023), the regret defined in 536 (3.2) can be bounded by 537

Note that in our selection of weight w t , it has two possible values. We decompose the summation 538 based on the two cases separately. We have 539

We consider J 1 , J 2 separately. For the term J 1 , we define Λ t = λ I + ∑ i ≤ t -1 ,w i =1 ( ϕ ( x i , a i ) -540 ϕ ( x i , b i ) )( ϕ ( x i , a i ) -ϕ ( x i , b i ) ) ⊤ . Then, we have Σ t ⪰ Λ t , and therefore 541

Using Lemma D.3 with x t = ϕ ( x t , a t ) -ϕ ( x t , b t ) , we have

For term J 2 , we note that w t < 1 implies that w t = α/ ∥ ϕ ( x t , a t ) -ϕ ( x t , b t ) ∥ Σ -1 t . Therefore, we 543 have 544

Using Lemma D.3 with x ′ t = √ w t ( ϕ ( x t , a t ) -ϕ ( x t , b t )) , we have 545

We conclude the proof of regret by combining (A.2) and (A.3).

B Proof of Theorems in Section 5

B.1 Proof of Theorem 5.3

In this subsection, we provide the proof of Theorem 5.3. We condition on the high-probability event in Lemma 5.1

Let r t = 2 r ∗ ( x t , a ∗ t ) -r ∗ ( x t , a t ) -r ∗ ( x t , b t ) be the regret incurred in round t . The following lemma provides the upper bound of r t .

Lemma B.1. Let 0 < δ < 1 . If we set β = √ λB + ( αC + √ d log((1 + 2 T/λ ) /δ ) ) /κ , on event E , the regret of Algorithm 1 incurred in round t can be upper bounded by

Moreover, the regret can be upper bounded by 555

With Lemma B.1, we can provide the proof of Theorem 5.3. 556

542

546

547

548

549

550

551

552

553

554

Proof of Theorem 5.3. Using Lemma B.1, the total regret can be upper bounded by 557

Our weight w t has two possible values. We decompose the summation based on the two cases 558 separately. We have 559

For the term J 1 , we consider a partial summation in rounds when w t = 1 . Let Λ t = λ I + 560 ∑ i ≤ k -1 ,w i =1 ( ϕ ( x i , a i ) -ϕ ( x i , b i ) )( ϕ ( x i , a i ) -ϕ ( x i , b i ) ) ⊤ . Then we have 561

where the second inequality holds due to Σ t ⪰ Λ t . The third inequality holds due to the Cauchy562 Schwartz inequality, The last inequality holds due to Lemma D.3. 563

For the term J 2 , the weight in this summation satisfies w t < 1 , and therefore w t = α/ ∥ ϕ ( x t , a t ) -564 ϕ ( x t , b t ) ∥ Σ -1 t . Then we have 565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

where the first equality holds due to the choice of w t . The first inequality holds because each term in the summation is positive. The last inequality holds due to Lemma D.3. Combining (B.1) and (B.2), we complete the proof of Theorem 5.3.

B.2 Proof of Theorem 5.5

Proof of Theorem 5.5. Our proof adapts the argument in Bogunovic et al. (2021) to dueling bandits. For any dimension d , we construct d instances, each with θ i = e i , where e i is the i -th standard basis vector. We set the action set A = { e i } d i =1 . Therefore, in the i -th instance, the reward for the i -th action will be 1. For the other actions, it will be 0. Therefore, the i -th action will be more preferable to any other action. While for other pairs, the feedback is simply a random guess. Consider an adversary that knows the exact instance. When the comparison involves the i -th action, it will corrupt the feedback with a random guess. Otherwise, it will not corrupt. In the i -th instance, the adversary stops the adversarial attack only after C times of comparison involving the i -th action. However, after Cd/ 4 rounds, at least d/ 2 actions have not been compared for C times. For the instances corresponding to these actions, the agent learns no information and suffers from Ω( dC ) regret. This completes the proof of Theorem 5.5.

581

582

583

584

585

586

587

588

589

590

591

592

B.3 Proof of Theorem 5.7

Proof of Theorem 5.7. Here, based on the relationship between C and the threshold ¯ C , we discuss two distinct cases separately.

In the scenario where ¯ C < C , Algorithm 1 can ensures a trivial regret bound, with the guarantee that Regret ( T ) ≤ 2 T .

In the scenario where C ≤ ¯ C , we know that ¯ C is remains a valid upper bound on the number of adversarial feedback. Under this situation, Algorithm 1 operates successfully with ¯ C adversarial feedback. Therefore, according to Theorem 5.3, the regret is upper bounded by

C Proof of Lemmas 5.1 and B.1

C.1 Proof of Lemma 5.1

Proof of Lemma 5.1. Using a similar reasoning in Li et al. (2017), we define some auxiliary quantities

In Algorithm 1, θ t is chosen to be the solution to the following equation, 593

Then we have 594

The analysis in Li et al. (2017); Di et al. (2023) shows that this equation has a unique solution, with 595 θ t = G -1 t ( Z t ) . Using the mean value theorem, for any θ 1 , θ 2 ∈ R d , there exists m ∈ [0 , 1] and 596 ¯ θ = m θ 1 +(1 -m ) θ 2 , such that the following equation holds, 597

We define F ( ¯ θ ) as 598

Moreover, we can see that G t ( θ ∗ ) = λκ θ ∗ . Recall Σ t = λ I + ∑ t -1 i =1 w i ( ϕ ( x i , a i ) -599 ϕ ( x i , b i ) )( ϕ ( x i , a i ) -ϕ ( x i , b i ) ) ⊤ . We have 600

where the first inequality holds due to ˙ µ ( · ) ≥ κ > 0 and F ( ¯ θ ) ⪰ κ Σ t . Then we have the following 601 estimate of the estimation error: 602

where the second inequality holds due to the triangle inequality and G t ( θ ∗ ) = λκ θ ∗ . The last 603 inequality holds due to Σ t ⪰ λ I . Finally, we need to bound the ∥ Z t ∥ Σ -1 t term. To study the impact 604 of adversarial feedback, we decompose the summation in (A.1) based on the adversarial feedback c t , 605 i.e., 606

When c i = 1 , i.e. with adversarial feedback, | γ i -ϵ i | = 1 . On the contrary, when c i = 0 , γ i = ϵ i . 607 Therefore, 608

Summing up the two equalties, we have 609

Therefore, 610

For the term I 1 , with probability at least 1 -δ , for all t ∈ [ T ] , it can be bounded by 611

due to Lemma D.2. Using w i ≤ 1 , we have w i ∥ ϕ ( x i , a i ) -ϕ ( x i , b i ) ∥ 2 ≤ 2 . Moreover, we have 612

617

618

619

620

621

622

where the first inequality holds because for every matrix A ∈ R d × d , det A ≤ ( Tr ( A ) /d ) d . The 613 second inequality holds due to √ w i ∥ ϕ ( x i , a i ) -ϕ ( x i , b i ) ∥ 2 ≤ 2 . Easy to see that det( Σ 0 ) = λ d . 614 The term I 1 can be bounded by 615

For I 2 , with our choice of the weight w i , we have 616

where the second inequality holds due to Σ t ⪰ Σ i . The third inequality holds due to w i ≤ α/ ∥ ( ϕ ( x i , a i ) -ϕ ( x i , b i )) ∥ ∥ Σ -1 i . The last inequality holds due to the definition of C . Combining (C.2) and (C.3), we complete the proof of Lemma 5.1.

C.2 Proof of Lemma B.1

Proof of Lemma B.1. Let the regret incurred in the t -th round by r t = 2 r ∗ ( x t , a ∗ t ) -r ∗ ( x t , a t ) -r ∗ ( x t , b t ) . It can be decomposed as

where the first inequality holds due to the Cauchy-Schwarz inequality. The second inequality holds 623 due to the high probability confidence event E . Using our action selection rule, we have 624

Adding the above two inequalities, we have 625

626

Therefore, we prove that the regret in round t can be upper bounded by

With a simple observation, we have r t ≤ 4 . Therefore, the total regret can be upper bounded by 627

628

D Auxiliary Lemmas 629

Lemma D.1 (Azuma-Hoeffding inequality, Cesa-Bianchi and Lugosi 2006) . Let { η k } K k =1 be a 630 martingale difference sequence with respect to a filtration {F t } satisfying | η t | ≤ R for some constant 631 R , η t is F t +1 -measurable, E [ η t |F t ] = 0 . Then for any 0 < δ < 1 , with probability at least 1 -δ , we 632 have 633

Lemma D.2 (Lemma 9 Abbasi-Yadkori et al. 2011) . Let { ϵ t } T t =1 be a real-valued stochastic process 634 with corresponding filtration {F t } T t =0 such that ϵ t is F t -measurable and ϵ t is conditionally R -sub635 Gaussian, i.e. 636

Let { x t } T t =1 be an R d -valued stochastic process where x t is F t -1 -measurable and for any t ∈ [ T ] , 637 we further define Σ t = λ I + ∑ t i =1 x i x ⊤ i . Then with probability at least 1 -δ , for all t ∈ [ T ] , we 638 have 639

Lemma D.3 (Lemma 11, Abbasi-Yadkori et al. 2011) . For any λ > 0 and sequence { x t } T t =1 ⊆ R d 640 for t ∈ [ T ] , define Z t = λ I + ∑ t -1 i =1 x i x ⊤ i . Then, provided that ∥ x t ∥ 2 ≤ L holds for all t ∈ [ T ] , we 641 have 642

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The primary contribution of this paper is addressing the challenge of adversarial feedback within the dueling bandit model, where feedback is represented as a binary preference label. Our research introduces a new perspective to machine learning. Unlike previous works on corruption-robust bandits, where corruption in each round affects the single-arm exploration and exploitation process. Flipping the preference label potentially impacts the expected reward of both actions chosen in a duel. This interaction can further affect subsequent decisions involving only one of these arms. Compared with previous adversarial dueling bandit work, we study the most direct label-flipping attack, which is aligned with many real-life preference-based learning scenarios. Our uncertainty-weighted maximum likelihood estimation method helps to solve this novel problem, in scenarios with known and unknown adversarial feedback. All the scope has been discussed clearly in our abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have added a Limitations setting in our main paper. We assume that the reward is linear with respect to some known feature maps. Although this setting is common in the literature, we observe that some recent works on dueling bandits can deal with nonlinear rewards (Li et al., 2024). Therefore, it's possible to extend our results to a more general setting. Another assumption concerns the lower bound of the derivative of the link function. Notably, in the logistic bandit model, which shares similarities with our setting through Bernoulli variables, some work (Abeille et al., 2021; Faury et al., 2022) can improve the dependency of κ from 1 /κ to √ κ . A similar improvement might be achieved in our setting as well.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be

used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have clearly stated and proved all the lemmas and theorems used in our theoretical results. To help readers understand the proof without checking all the details, we provide a roadmap of our proof in Appendix A. We also write explanation and clarification for every formula in our paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper is mainly theoretical but we also do numerical experiments to justify the correctness of our results. We provide all the information to reproduce our results in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case

750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803

of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our experiments involve synthetic data generated from a generalized linear model, which is quite simple and easy to reproduce. That's why we do not provide access to our data and code. All the information required to reproduce the results is provided in Section 6. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We only have synthetic experiments and it can be reproduced on CPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper studies contextual dueling bandits with adversarial feedback. Our primary objective is to propel advancements in bandit theory by introducing a more robust algorithm backed by solid theoretical guarantees. The uncertainty-weighted approach we have developed for dueling bandits holds significant potential to address the issue of adversarial feedback in preference-based data, which could be instrumental in enhancing the robustness of generative models against adversarial attacks, thereby contributing positively to the societal impact and reliability of machine learning applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work studies linear contextual dueling bandits with adversarial feedback. In each round $t$ the agent observes a context $x_t$ and chooses two actions $(a_t,b_t)$. The environment generates a binary preference label $\\ell_t = \\mathbb{I}(a_t > b_t)$. The underlying assumption is that there exists a linear reward function $r(x,a) = \\theta_{\\star}^{\\top}\\phi(x,a)$, where $\\theta_*$ is a latent $d$-dimensional vector and $\\phi$ is a known feature map such that $\\|\\|\\theta_*\\|\\|_2 \\le B$ and $\\|\\|\\phi\\|\\|_2 \\le 1$.\n\nBased on this, the preference $\\ell_t$ is a random variable such that\n$$\n\\mathbb{P}\\big(a > b \\mid x\\big) = \\sigma\\big(r(x,a)-r(x,b)\\big)\n$$\nThe link function $\\sigma$ is antisymmetric, $\\sigma(-z) = 1-\\sigma(z)$ and such that $\\sigma' \\ge \\kappa > 0$.\n\nIt is further assumed that a nonoblivious adversary may occasionally flip the preference label with the knowlege of $(a_t,b_t)$, where $C_T$ indicates the number of flips in the first $T$ rounds.\n\nThe regret is measured according to the following formula\n$$\n\\max_a \\sum_t 2r(x_t,a) - \\sum_t \\Big( r(x_t,a_t) + r(x_t,a_t) \\Big)\n$$\nThe main result is a regret bound of the order $d\\sqrt{T} + dC$, ignoring logarithmic factors. This bound is tight because $\\Omega(d\\sqrt{T})$ is the known lower bound without adversarial corruption and $\\Omega(dC)$ is shown to be the regret due to the adversarial corruption.\n\nExperiments complete the contribution.\n\nSTRENGTHS:\nThe topic is of adversarial feedback in dueling bandits is interesting and was only studied in a $K$-armed setting.\n\nThe regret bound is tight up to log factors\n\nThe related work section is complete and accurate.\n\nThe experimental section is significant.\n\nWEAKNESSES:\nThe main results (upper and lower bounds) appear to be mostly based on combinations of known techniques from previous papers. The originality of the technical contributions is unclear.\n\nThe analysis of the $C$ unknown case (Section 5.2) is trivial.\n\nAssumption 3.2 could create bad dependencies in the bounds on $B$ (and $\\|\\|\\theta_{*}\\|\\|_2$).\n\nThe conditions $\\sigma' \\le 1$ and $\\phi_t^{\\top}\\theta_{*} \\le 1$ in the paragraph before (4.3) are not explicitely stated in the assumptions. Moreover, the second condition seem to imply that $B \\le 1$.\n\nThere is no discussion on the hardness of computing $(a_t,b_t)$.\n\nQUESTIONS:\n* Can the authors point out what where the main technical challenges in the proof of Theorem 5.3?\n* Please compute explicit values for $\\kappa$ in terms of $B$ and $\\|\\|\\theta_{*}\\|\\|_2$ for concrete choices of $\\sigma$.\n* Please elaborate on the hardness of computing $(a_t,b_t)$.\n\nLIMITATIONS:\nThere is an explicit limiation paragraph in the conclusions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigated the contextual dueling bandits with adversarial feedback, where the adversary can corrupt the binary feedback of the agent to a certain level. A new algorithm named RCDB has been proposed. The key idea lies in the utilization of uncertainty-weight MLE. Regret analysis of RCDB was provided along with some experimental evaluations.\n\nSTRENGTHS:\n- This paper studies a known problem but with new angle, i.e., the adversary can corrupt the binary feedback of the agent to a certain level. The problem is well motivated. \n- A novel algorithm named RCDB was designed and incorporated uncertainty-dependent weighting into the MLE.\n- The theoretical performance of RCDB in terms of regret is provided. \n- Experimental results to validate the performance of RCDB is presented and compared to existing methods.\n\nWEAKNESSES:\n- Assumption 3.1 assumes a linear reward. The reviewer agrees that this is a \"widely-used\" assumption in the recent RLHF literature, but was curious if your framework and regret analysis can be extended without such an assumption? If not, what are the new challenges? Can you comment on this? \n- The construction of the parameter estimator of $\\theta$ requires the Taylor expansion. How the $\\approx$s in Section 4 impact the regret analysis? \n- The experiments were run for multiple times, however, the variance is not shown in Figure 1. \n- The paper is very dense, and the authors have changed the template a bit, e.g., the space has been largely squeezed throughout the paper.\n\nQUESTIONS:\n- Assumption 3.1 assumes a linear reward. The reviewer agrees that this is a \"widely-used\" assumption in the recent RLHF literature, but was curious if your framework and regret analysis can be extended without such an assumption? If not, what are the new challenges? Can you comment on this? The reviewer noticed that the authors discussed this at the end of the paper and mentioned Li et al. 2024. Will such an extension be straightforward? \n- The construction of the parameter estimator of $\\theta$ requires the Taylor expansion. How the $\\approx$s in Section 4 impact the regret analysis? \n- The experiments were run for multiple times, however, the variance is not shown in Figure 1. \n- The paper is very dense, and the authors have changed the template a bit, e.g., the space has been largely squeezed throughout the paper.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The author proposed an algorithm, coined robust contextual dueling bandits (RCDB) for advarial feedback, using uncertainty-weighted maximum likelihood estimation. The algorithm guarantees $\widetilde{O}(d\sqrt{T}+CT)$.

STRENGTHS:
1. Their algorithm is not limited to a finite number of arms.

2. Their algorithm considers adversarial attacks based on selected actions (although the maximum number of adversarial attacks is restricted).

WEAKNESSES:
1. Hard to follow the paper

QUESTIONS:
1. $\Sigma_t$ must be the Gram matrix in line 209, $\Sigma_t$ appeared without introducing.

2. Please specify exactly where in the supplementary materials proofs of these results (Lemma 5.1 ~ Theorem 5.7) are located.

3. It seems to specify the initial weight $w_t$ in Algorithm 1. $w_t$ is specified in Line 8 in Algorithm 1, but it is used from line 3.

4. Further explanation is needed for lines 204-209. For instance, clarify how the property of  $\mathcal{F}_t$-measurability is used in the conditional expectation in line 207. Additionally, explain why the assumption  $|\theta_t-\theta^*|<<1$ is made to consider Taylor's expansion.

5. It would be better to clearly state that the proposed algorithm is effective under both known and unknown numbers of adversarial attacks. Prior to Section 5, it is unclear whether this paper considers both aspects or only one of them.

LIMITATIONS:
1. The authors precisely pointed out the limitations of their results, such as the reward function being linear with a known feature map.

2. Additionally, choosing $a_t, b_t$ (by computing argmax) might be infeasible when interaction with the environment needs to be fast.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
This paper studies the Contextual Dueling Bandits from Adversarial Feedback problem, in a linear reward setting. The authors propose an algorithm named robust contextual dueling bandits (RCDB), which is designed based on uncertainty-weighted regression and MLE. The authors prove that the proposed algorithm achieves a nearly optimal regret upper bound that matches the lower bound both in scenarios with and without (C=0) adversarial feedback. Experimental evaluations are provided to validate the theoretical results.

STRENGTHS:
1. The problem is well-motivated and important.
2. The paper is well-written.
3. The authors prove a nearly optimal regret upper bound for the proposed algorithm that matches the lower bound with and without (C=0) adversarial feedback. 
4. The authors also conduct some experiments to validate the theoretical results.

WEAKNESSES:
1. The title may be a little bit misleading, I think the setting of this paper is the adversarial corruption setting, not the setting with completely adversarial feedback. And the setting is the linear reward model, which is not specified in the title.

2. I have not checked the details, but I feel the uncertainty-weighted technique (which is the key to dealing with the corruption in the linear bandits) is mostly based on the previous works, could the authors highlight the technical difficulties in the dueling bandit setting?

QUESTIONS:
See the weakness above.

LIMITATIONS:
No negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12