Efficient Adversarial Attacks on High-dimensional Offline Bandits
Authors: Seyed Mohammad Hadi Hosseini
Amir Najafi
Mahdieh Soleymani Baghshah
Venue: ICLR 2026 Poster (2026)
Keywords: Offline Bandits
Adversarial Attacks
Primary area: learning theory
TL;DR: We present a method demonstrating that small, targeted perturbations to reward models can hijack offline bandit trajectories, particularly in high-dimensional settings.

Abstract: Bandit algorithms have recently emerged as a powerful tool for evaluating machine learning models, including generative image models and large language models, by efficiently identifying top-performing candidates without exhaustive comparisons. These methods typically rely on a reward model---often distributed with public weights on platforms such as Hugging Face---to provide feedback to the bandit. While online evaluation is expensive and requires repeated trials, offline evaluation with logged data has become an attractive alternative. However, the adversarial robustness of offline bandit evaluation remains largely unexplored, particularly when an attacker perturbs the reward model (rather than the training data) prior to bandit training. In this work, we fill this gap by investigating, both theoretically and empirically, the vulnerability of offline bandit training to adversarial manipulations of the reward model. We introduce a novel threat model in which an attacker exploits offline data in high-dimensional settings to hijack the bandit's behavior. Starting with linear reward functions and extending to nonlinear models such as ReLU neural networks, we study attacks on two Hugging Face evaluators used for generative model assessment: one measuring aesthetic quality and the other assessing compositional alignment. Our results show that even small, imperceptible perturbations to the reward model’s weights can drastically alter the bandit's behavior. From a theoretical perspective, we prove a striking high-dimensional effect: as input dimensionality increases, the perturbation norm required for a successful attack decreases, making modern applications such as image evaluation especially vulnerable. 
Extensive experiments confirm that naive random perturbations are ineffective, whereas carefully targeted perturbations achieve near-perfect attack success rates. 
To address computational challenges, we design efficient heuristics that preserve almost 100\% success while dramatically reducing attack cost. In parallel, we propose a practical defense mechanism that partially mitigates such attacks, paving the way for safer offline bandit evaluation. Finally, we validate our findings on the UCB bandit and provide theoretical evidence that adversaries can delay optimal arm selection proportionally to the input dimension. Code is available at the anonymous repository: [https://anonymous.4open.science/r/offline-bandit](https://anonymous.4open.science/r/offline-bandit).

Published as a conference paper at ICLR 2026
EFFICIENT ADVERSARIAL ATTACKS ON HIGH-DIMENSIONAL OFFLINE BANDITS
Seyed Mohammad Hadi Hosseini, Amir Najafi & Mahdieh Soleymani Baghshah Department of Computer Engineering
Sharif University of Technology
(hadi. wws.oseeenne ni17, rr.ii.n nainajaff ji sol solleymai
ABSTRACT
Bandit algorithms have recently emerged as a powerful tool for evaluating ma- chine learning models, including generative image models and large language models, by efficiently identifying top-performing candidates without exhaustive comparisons. These methods typically rely on a reward model-ofter often distributed with public weights on platforms such as Hugging Face-to provide feedback the bandit. While online evaluation is expensive and requires repeated trials, of- fline evaluation with logged data has become attractive alternative. However, the adversarial robustness of offline bandit evaluation remains largely unexplored, particularly when an attacker perturbs the reward model (rather than the training data) prior to bandit training. In this work, we fill this gap by investigating, both theoretically and empirically, the vulnerability of offline bandit training adver- sarial manipulations of the reward model. We introduce a novel threat model in which an attacker exploits offline data in high-dimensional settings to hijack the bandit's behavior. Starting with linear reward functions and extending non- linear models such as ReLU neural networks, we study attacks on two Hugging Face evaluators used for generative model assessment: quality and the other assessing compositional alignment. Our results show that even small, imperceptible perturbations to the reward model's weights can dras- tically alter the bandit's 's behavior. From a theoretical perspective, we prove a striking high-dimensional effect: as input dimensionality increases, the perturba- tion norm required for a successful attack decreases, making modern applications such as image evaluation especially vulnerable. Extensive experiments confirm that naive random perturbations are ineffective, whereas carefully targeted pertur- bations achieve near-perfect attack success rates. To address computational chal- lenges, we design efficient heuristics that preserve almost 100% success while dra- matically reducing attack cost. In parallel, we propose practical defense mech- anism that partially mitigates such attacks, paving the way for safer offline bandit evaluation. Finally, we validate our findings on the UCB bandit and provide theo- retical evidence that adversaries can delay optimal arm selection proportionally the input dimension. Code is publicly available at: https: / github.. com/ hadi-hor ces---osseinnn atweeiiidaaveesiiaal... attacks offline bandits.
measuring aesthetic
INTRODUCTION
The multi-armed bandit (MAB) framework is a cornerstone of online learning. balaneing explo- ration and exploitation to efficiently identify the optimal choice (or "arm") within limited time horizon. MAB algorithms have been widely applied in domains such as online recommendation sys- tems (Kveton et al., 2015; Chapelle et al., 2014; Li et al., 2010), adaptive clinical trials (Kuleshov & Precup, 2014), and dynamic pricing (Tullii et al., 2024). More recently, they have been adopted for evaluating generative
models-fronm diffusion models (Rezaci et al., 2025; Hu et al., 2025; 2024b; Ho et al., 2020; Rombach et al., 2022) and GANs (Rezaei et al., 2025; Hu et al., 2024a; Karras et al., 2019) to large language models (LLMs) such ChatGPT (Hu et al., 2025; Dai al., 2025; Achiam et al., 2023), Gemini (Hu et al., 2025; Dai et al., 2025; Team et al., 2023), and DeepSeck (Hu et al., 2025; Guo al., 2025). In this setting, bandit algorithms (e.g., Upper Confidence Bound (UCB)

---

Published as a conference paper at ICLR 2026
from Auer (2002)) offer sample-efficient alternative naive pairwise comparisons by rapidly identifying the best-performing model.
While online evaluation remains effective, it is often costly and impractical for rapidly evolving gen- crative models. As a result, practitioners increasingly turn to offline evaluation, where a fixed logged dataset is reused to compare algorithms (e.g., UCB, ETC (Garivier et al., 2016), e-greedy (Dann et al., 2022)) or tune hyperparameters without repeated data collection. Despite its practicality, this setting introduces two overlooked risks, First, it vulnerable to adversarial manipulation that could bias the selection process. Second, it is prone to overfitting to the reward model when multiple degrees of freedom are available. For instance, recent work has shown a tendency to test numer- ous performance metrics or continuously adjust metric hyperparameters on the same logged dataset when training MABs (Rezaei et al., 2025; Hu et al., 2024b; Saito & Nomura, 2024).
We aim to provide partial answers to the above concerns across range of settings. Specifically, we study adversaries that manipulate only the reward model (rather than the data samples) in order to hijack the behavior of bandit algorithm. While prior work has examined training-time adversarial attacks on bayicls-typiccaly by altering reward values or samples during online training (see Ap- pendix A)-no prior research has considered attacks on the reward model before training. We fill this gap by showing that even a weak adversary can succeed in this setting when the data is high- dimensional. In our threat model, the adversary has access to the offline (logged) dataset but cannot modify the samples, nor can it interfere with the learner's training procedure; its only leverage is to perturb the reward model prior to training.
We analyze two reward-moder scenarios. First, we study linear reward functions in sigh-dimensional spaces. This setting, though simplified, offers clarity and valuable insights into the mechanics of our attack. Our results show that even small perturbations to the weights of a linear model can dramatically alter the bandit's behavior. Second, we extend our analysis nonlinear reward models such as deep neural networks (LeCun et al.. 2015), which are widely used in practice. For instance, modern image-based generative models are often evaluated with neural reward functions such as CLIP (Radford et al., 2021: LAION-AI, 2021), BLIP (Li et al., 2022; Xu et al., 2023), and VQA models (Hu et al., 2023; Singh & Zheng, 2023; Cho et al., 2024). These models are typically open-source, with publicly available weights on platforms such as Hugging Face-making them convenient, but also vulnerable. We select two real-world image reward models from Hugging Face and apply our attack to them: one for assessing compositionality alignment and the other for evaluating aesthetic quality. We demonstrate that exposing verifier weights can be exploited: by constructing linear approximation of the reward model, an adversary can craft perturbations of imperceptibly small norm that nonetheless yield highly effective attacks. Moreover, we show that this linear approximation becomes increasingly accurate as the network's hidden layer width grows, consistent with predictions from Neural Tangent Kernel (NTK) theory (Jacot et al., 2018; 8; Golikov al., 2022). We identify several key insights and introduce novel algorithms that enable highly effective attacks on offline bandit training, supported by both theoretical analysis and empirical validation. Our main contributions are:
Theoretical foundations. We show that effective attacks require solving a targeted optimization problem; naive random perturbations fail to compromise the reward model. Empirically, we ob- serve an inverse relationship between dimension and attack cost: as input dimensionality increases, the required perturbation norm decreases, making high-dimensional data (c.g., images) especially vuinerable. We then complement empirical observations via theoretical support. Formally, for K-armed bandit over a d-dimensional input space with horizon T, we prove that full-trajectory attacks are feasible whenever d 2 KT (Theorem 3.3), and that the fy-norm of the required pertur- bation shrinks as ((d-1/2) (Theorem 3.4). Our analysis is based on tools such as high-dimensional concentration inequalities and random matrix theory.
Computational efficiency. Solving the full optimization problem hijack an entire trajectory can be computationally expensive. We design heuristic methods that dramatically reduce this cost while maintaining near-perfect attack success rates. We validate these attacks across multiple bandit algorithms, focusing on UCB but also extending to ETC and e-greedy, showing that our method generalizes beyond a single algorithm.
Synthetic and real-world evaluation. We first validate our approach on synthetic data, then extend to real-world settings by targeting two widely used Hugging Face evaluators for generative models:
2

---

Published as a conference paper at ICLR 2026
measuring aesthetic quality and the other compositional alignment. In both cases, attacks remain highly effective.
Defense. We propose simple defense mechanism that partially mitigate attacks under certain conditions, reducing attack success rates. However, a complete defense remains open, highlighting an important direction for future research.
2 PROBLEM DEFINITION AND NOTATION
For K E N, let [K] = 21,2.... K}. We consider a stochastic multi-armed bandit problem with K arms, where each arm [K] associated with a data tata-generating distribution P; over R.d Here, d denotes the dimension of the input space. At each round, when the bandit pulls arm i, it receives a sample X ~ P. drawn independently from past rounds. The reward is then computed as r(X), where r IR + R is known reward function that may be perturbed adversarially before training begins. We study two instantiations the reward function:
1. Linear model: r(X) w X, with fixed parameter vector E R..
2. Neural network model: r(X) NN,(X), where NNo feedforward network with parameters A E Te and some fixed activation function a--) R - R. The parameter
dimension D d W1 + W1 W2 + ................................ WT-1" WL + WL; for network with hidden layers of widths W1, W ... Wt. respectively.
Throughout, we focus on the high-dimensional regime: either >> (linear case) or max; W, >> (neural network case). We assume arm i" E [K] is optimal, i.c., Ep:.Ir(X) > Ep [X)]] for i * i", where E[-] is expectation operator. Hence, we expect the bandit algorithm to (with high probability) learn to pull the optimal arm i" after some number T 1 of exploratory trials across the K arms. The value T is referred to as the horizon. Suppose we are given K logged datasets consisting of independent samples from each arm distribution PI...., Pk. Specifically, let D; (x,9 Ij e [nw]}, be a dataset of size n,2 containing i..d. samples from P.. When the bandit pulls
arm for the t,-th time, it receives the sample x for t; E [n,]. We assume that each dataset Di.k contains sufficiently many samples that, throughout the T rounds, the bandit always observes fresh sample upon each arm pull.
Our goal is to design adversarial attack the reward function r(-) such that, with high probability, the bandit fails to identify and exploit the optimal arm. Assume an adversary a with access the offline logged datasets D,,... DR. and the ability perturb the reward model r(-). In the linear case, the adversary modifies the parameter w to w +- for some perturbation vector 5 Rd the neural network case, the adversary modifies the parameter vector 0 +8, where 5 E D. In both settings, the perturbation norm is chosen small as possible while still guaranteeing a prescribed level of damage the bandit's 's trajectory, for example, ensuring that the number of times the optimal arm i" is pulled falls below some threshold T' < T. In stronger attack scenarios, dd may even enforce pre-specified target trajectory for the multi-armed bandit.
Our Problem:
METHOD
In this section, focus UCB (Auer, 2002) algorithm for simplicity, but our findings extend naturally to ETC and s-greedy, as detailed in Appendix D. UCB begins by pulling each arm once, and then proceeds by selecting arms in a way that balances exploitation and exploration: it favors arms with higher observed average rewards, while a carefully designed exploration term encourages underexplored arms to be pulled. It is theoretically known that UCB achieves a regret bound of (log T',, which is optimal up to constants, making it widely used in practice. Let A, A,))) for I E [T] denote the arm chosen step when the adversary e selects the perturbation vector 8. Then, the algorithm proceeds for 1. = 1...., T as follows:
. K, pull each arm once. That is, A For L 1,...,
3

---

Published as a conference paper at ICLR 2026
For t K, select the next arm according to the following deterministic maximization:
N.(0)
2 N((t)) f
(0) - arg max
Ac(::)
1 - N.(O) _=1
Σ
5) +
(I)
kE(K]
where (t) denotes the number of times arm has been pulled up step 6, and r(X; 8) denotes the adversarially perturbed reward value of input X given the perturbation vector 8, e.g., rXx; 6) = (w +ō) X in the linear case. Also, (i; 5) is called the UCB score of arm at time step
The adversary aims keep ||6|| 22 as small as possible. However, depending on its objective, a may design ò to achieve one of the following goals:
Full Trajectory Attack. In this scenario, attempts to force the bandit to follow a completely predetermined, arbitrary, and fixed target trajectory A, E [K] for all i E [7]. This requires solving the following constrained optimization problem:
8 & arg min | subject to Ar(A(;;) (d;
(2)
The optimization in equation 2 requires - K)(KK(K - 1) constraints. In the linear reward model, these constraints are linear in 8.
Trajectory-Fe Attack. A less restrictive attack requires the bandit to avoid pulling the optimal arm at given of time-steps T [T). Note that T may include all steps from K to T. Unlike the full trajectory attack, here e only ensures that i" is not selected, without prescribing exactly which arm must be pulled. One simplified formalization is as follows: consider an arbitrary trajectory A, E [K] for all
[T] such that A, # i*. Then we can define:
8 A aig | subject to Ar(ā;;)) 5))
(3)
5
Here, the auxiliary sequence As * i' for K < < t is arbitrary and can be optimized (c.g., via heuristics) by the attacker to further reduce attack strength. In our experiments, we use simple round-robin strategy for A. The formulation in equation 3 requires only T K constraints, suming T includes all steps from K + 1 to T. Trajectory-five attacks thus reduce the number of constraints by a factor of K- 1, significantly lowering computational cost when K is large. How- ever, for large T, they may still be expensive and difficult to design. To address this, we propose an online attack strategy.
Online Score-Aware (OSA) Attack. Consider fixed set of time-steps T C [T]. We iterate over t E in increasing order. Let 8;_1 denote the optimal attack up to time step At each step i, the attacker checks whether the UCB score Az(i"; ;;;1) is the largest among all arms. If not, we simply move on to the next l. Otherwise, we add the constraint
ArGr; 8) > (i);)))
where is is the index of the runner-up arm with the second highest UCB score at time The solution of is then updated using the additional constraint, and the process continues until the end of T. Therefore, instead of single QP, a series of QPs with increasing number of constraints should be solved. However, the number of constraints in each QP is usually very small and the problem can be solved almost instantly.
While a full theoretical analysis of the OSA attack is beyond the scope of this paper, we empirically observe that it substantially reduces computational cost. practice, the number of effective con- straints generated by the online procedure is far smaller than T of-oftn resembling Olog which leads to significantly faster attack construction compared to both full-trajectory and trajectory- methods. Pseudocode for all methods is provided in Appendix K. Some interesting and fundamental questions arise regarding the attack designs discussed above:
Optimization formulation. How can these attack designs be explicitly instantiated real- world applications? In the next section, we show that for linear reward model r(X)
4

---

Published as a conference paper at ICLR 2026
w X, all of the above formulations reduce to well-structured convex Quadratic Programs (QPs), which can be solved efficiently. In this setting, reducing the number of constraints substantially accelerates optimization. Interestingly, similar argument also extends to wide neural network models.
Feasibility, The efficiency gains above are only meaningful the optimization problems are feasible, i.e., if there exists at least one perturbation vector ó satisfying all constraints. In Section 3.2, we provide theoretical guarantees showing that, under mild assumptions the reward model and the Gata-generating distributions PI..... PK. the constraint set is feasible with probability 1, provided that the data dimension d (or, in the case of neural networks, the maximum width max; W,) exceeds the number of constraints. Moreover, we establish a second main result: the norm of feasible full-trajectory attack decreases at least as 1/2) with dimension d.
3.1 OPTIMIZATION
The following result shows that, under a linear reward model, the proposed attack formulations admit an efficient convex optimization representation.
Theorem 3.1 (Linear Reward Model). Consider the three atlack designs in Section 3 under the linear reward model r(X) T QTg X for some fixed vector w E Rd Then there exist an indexed vector set {T and a scalar set { ). i [K -1] and K < EtST. such that all three attack
designs can be expressed as the quadratic program
8 min ||0113
subject to TT, RA,
K (T].
(4)
The values Ti and Rs, are determined solely by the offline datasets Di.... Dr and the original reward model w. The joint arm-timestep index set I depends only on the choice of attack type (full- trajectory, trajectory-free, or OSA) and is determined according to the specification of the method.
Proof is given in Appendix B. The computational cost is governed by the size of I. In particular, for full-trajectory attacks we have [I] O(TK), for trajectory-free attacks [Z] -T),,, and for OSA attacks the effective size of I is typically much smaller in practice. Solving a QP of the form equation 4 with d variables and [Z] constraints requires approximately ( + d operations (Gay et al., 1998), highlighting the central role of the constraint set size the overall complexity. In our experiments (see Figure 7), we observed that OSA can reduce [Z] to as small as O(log T").
When the reward model is neural network NNp(.), the UCB score (4 5) equation linear in 6, and the convex reduction in Theorem 3.1 no longer holds. However, results from neural tangent kernel (NTK) theory (Jacot et al., 2018) show that for sufficienlly wide networks with ran- domly initialized weights, the network behaves approximately linearly in its parameters. Formally,
non-
NNo 5 (X) (X) + (X)N 8 +O(W max
V Rd
(5)
where W max denotes the network width. Thus, for highly overparameterized networks the reward model can be well approximated by an affine function of 6, yielding quadratic program analogous to equation 4, with afiine constraints. Recall that NoNNo represents the embedding of the input X in the neural tangent space of the network NNo, centered at the parameter vector O (Jacot et al., 2018).
fact, the output of an overparameterizedd neural network, for any fixed input X, asymptotically mimics a linear reward function with respect the perturbation vector ô d-essentially reducing to the purely linear case.
Corollary 3.2 (Overparameterizeed Neural Networks). For asymptotically wide neural networks, linearization via the NTK approximation reduces the attack design to convex quadratic program with D variables as Theorem 3.1, where is the number of parameters in O. This program can be solved S + D) operations.
3.2 THEORETICAL GUARANTEES: FEASIBILITY AND ATTACK NORM
In this section, present two sets of theoretical guarantees. The first, stated in Theorem 3.3, estab- lishes that under mild "non-degeneracy" conditions on the data- tata-gennrating distributions Pi....,P5 PR. and in sufficiently high-dimensional regime (i.e., d > TK), there always exists attack (with
5

---

Published as a conference paper at ICLR 2026
probability over the sampling of datasets D2) that can steer the bandit toward any desired trajec- tory.
Theorem 3.3 (Feasibility Guarantee). Assume the data-generating distributions Pi..... Px are non-singular, Le.,
Amin (Cov(P.)) 00,
where Cov(.) denotes the x d covariance matrix and Amin ( the minimum eigenvalue. If > (T- K)(K - -)), then with probability all optimization problems corresponding to the full-trajectory attack, the trajectory-free attack, and the OSA attack are feasible.
The proof (Appendix B) relies on linear-algebraic arguments and anti-concentration properties of non-degenerate distributions. A stronger result, requiring more technical machinery, is given in Theorem 3.4, where we show that in the high-dimensional regime, and under stronger assumptions on PI...., Pk. the magnitude of the attack decreases as (dt-1/2 with dimension. We conjecture that an even more general result could be established for near-product measures, which cover broader class distributions, though this lies beyond the scope of the present work.
For concreteness, let μ4; Ep, [ for E [K]. and assume ||/4:12 1 for all í. For the linear reward model, we further assume | 1, with u,,. a configuration arises naturally in high-dimensional spaces, where independently drawn random vectors tend to be nearly orthogonal. We emphasize that these assumptions are introduced solely to simplify notation and exposition in subsequent sections; they are not essential to our analysis and do not affect the algorithmic design.
and μt, w for i * i.. Such
Theorem 3.4 (f2-norm of High-dimensional Attack). Consider the assumptions mentioned above for d >> 2. For simplicity, assume all PI...., Pk product measures and Cov(P.) I for all i E [K]. Then, provided d 2 KT, the full-trajectory attack of equation satisfies
(V
(T log T log d
T3
118" I
(6)
Kd
with probability at least 1 - 2d over the randonuess of Di...., DK.
The full proof, deferred Appendix B, leverages tools from linear algebra, optimization, and ran- dom matrix theory. Here we provide a simplified proof sketch illustrating why attacks on high- dimensional bandits are feasible. It is well known that estimating the mean of high-dimensional random vectors becomes increasingly difficult as the dimension grows: fluctuations across many co- ordinates accumulate, making accurate estimation unreliable. A multi-armed bandit can be viewed as repeatedly estimating the means of several high-dimensional distributions and then selecting the arm that maximizes its reward. When the number of observations per arm, roughly T/K, is much smaller than the dimension d, such estimates are inherently unstable. In this regime, even a sim- ple manipulation of the reward function when has sufficient degrees of freedom, as in linear models or overparameterized neural networks-con easily mislead the bandit. Theorem 3.4 further establishes that the required attack magnitude actually decreases as the dimension increases.
4 EXPERIMENTS
In this section, we conduct series of experiments to assess the effectiveness of our proposed attack method. Our study focuses on some central questions: (Q1) How does the attack's 's performance differ between linear and non-linear reward models with synthetic data? (Section 4.1). (Q2) What
the impact of dimensionality on attack performance? (Section 4.2). (Q3) Can our method be applied to weal-world settings, including real data and reward models? (Section 4.3). (Q4) Can our attack be applied to bandit algorithms other than UCB? (Section 4.4). (Q5) Are there any defenses that can
prevent this attack under certain circumstances? (Section 4.5). (Q6) Can random noise perturbations
achieve results comparable to method? (Section 4.6). (Q7) How does increasing the width of
the non-linear reward model affect the attack success rate? (Section 4.7). Q8 How effective is our method against robust stochastic bandit algorithms under reward corruption? (Section 4.8)

---

Published as a conference paper at ICLR 2026
Linear Reward Model
Non-linear Raward Model
314
Attack Time (secondr))
223
Attack (saconde)
Ko3, de10000
Nv3, de
Kas, d=10000
Kes, de de
Ked. mAn mAA
K+3, mox 0000
WwwsW-200 2009
K.5. mox 0600
Figure 1: Comparison of attack performance on linear and nonlinear reward models using OSA, Trajectory- Free, and Full Trajectory methods. All attacks achieve 100% success rate. The attack times (in seconds) are displayed on a logarithmic scale in the plot, highlighting the superior effi ciency of OSA and illustrating the effect of varying parameters K and d (or max; W2) on attack duration.
4.1 SYNTHETIC SETTING: ATTACK PERFORMANCE ACROSS REWARD MODELS
In this experiment, we evaluate the performance of our attack on reward models, examining its effec- tiveness in both linear and non-linear settings for synthetic data. To provide a more comprehensive assessment, we vary key parameters such as K and d. Furthermore, We evaluate all variants of our method: OSA, Trajectory--Fre and Tull-Trajectory.
This experiment focuses on two aspects: (1) demonstrating that both linear and non-linear reward models can be successfully attacked, and (2) comparing the attack time of the OSA and Trajectory methods. As shown Figure 1, all variants of our attack achieve a 100% attack success rate (ASR) on the UCB algorithm. From these results, we observe that the OSA method substantially reduces the time required to generate perturbations (shown in the logarithmic scale). This demonstrates the clear efficiency advantage of the OSA approach. We also present additional experiments in Appendix C, showing that the number constraints OSA scales as Ollog 7).
4.2 IMPACT OF DIMENSIONALITY ON ATTACK PERFORMANCE
In this experiment, we validate our claim that increasing input dimensionality makes the bandit algo- rithm vulnerable to attacks. To investigate this, we vary the input dimensionality while keeping all other parameters fixed. We observe that both the & and for norms the perturbation decrease rapidly, indicating that in higher-dimensional settings, smaller perturbations suffice successfully attack the algorithm. Additionally, we reformulate the optimization problem from a Feasibility Pro- gram (FP) to a Quadratic Program (QP) to evaluate how effectively our method approximates the optimal solution. In the oow-dimensional regime, the OSA solution shows a small gap relative to the Optimal solution. However, as the input dimensionality increases, this gap diminishes, demon- strating that our OSA method achieves near-optimal attacks in higher-dimensional settings. We can conclude that our OSA method not only identifies a perturbation quickly, but also produces one that very close to the optimal solution. Moreover, investigate whether our attack in high-dimension imperceptible or not based on its geometry Appendix H.
4.3 REAL-WORLD SETTING: REAL DATA AND REWARD MODELS
We demonstrate that our attack can be applied real-world settings, including real data and re- ward models, using image generative models as case study. Specifically, we evaluate attack on five generative models: Stable Diffusion 3 (SD3) and Stable Diffusion 1.4 (Rombach et al., 2022), Kandinsky (Vladimir al., 2024), Openjourney, and Stable Diffusion XL (Podell et al., 2023). We sample 30 random prompts from GenAl-Bench (Li et al., 2024) and curate logged data for each prompt by generating outputs from the models using seeds ranging from to 100. For

---

Published as a conference paper at ICLR 2026
Norms Acress Dimensions
ASR Beforemiter Defense
Nomo 271
0.25
Mer 12F1
0.20
Teartrry /dbert
0.15
Marm
28
0.30
Dimans on (D)
Figure 2: Left: Effect of increasing input dimensionality on attack magnitude. Both the & and
lo norms of the perturbation decrease as the input dimensionality increases, indicating that higher- dimensional inputs are more vulnerable to attacks. Experimental settings: T 100. = 3, ASR= 100%. Right: Effect of applying the proposed defense attack success rates. Shuffling portion of the logged data before running the bandit algorithm significantly reduces the effectiveness of attacks. Specifically, shuffle the first T 7/2 of the logged data before execution.
reward models, we use two widely adopted models: Image Reward (Xu et al., 2023), which eval- uates the compositional alignment between the generated image and its prompt, and the Aesthetic Model (LAION-AI, 2021), which assesses the aesthetic quality of the generated image. Both models are widely used in prior work, and their weights are publicly available HuggingFace. Moreover, both models are large, so we keep their weights frozen, except for a small subset used in the attack. As shown in Figure 3, the distribution of our attack success rate (ASR) is high: if we select random prompt, generate logged data for it, and perform our attack, there is approximately an 80% probabil- ity that the ASR falls between 90-1009 We further demonstrate that method can also attack randomly initialized Reward model with arbitrary weights; details are provided in Appendix F. Ad- ditionally, illustrate our method more clearly, complete and detailed visualization of our attack on the Aesthetic Reward Model is provided Appendix J.
Distribution Aesthatic Reward Hodel
Distribution ASR vature
Reward Model
ASR
(a) Aesthetic Reward Model
(b) Image Reward Model
Figure 3: Distribution of ASR values for the Aesthetic and Image reward models under the OSA method across two configurations. Each subplot shows the ASR distribution for one reward model.
4.4 ATTACKING BANDIT ALGORITHMS BEYOND UCB
Throughout this paper, we use the UCB (Auer, 2002) algorithm evaluate the robustness of our attack, we also consider scenarios where the adversary employs other bandit algorithms, specifically ETC (Garivier al., 2016) and e-greedy (Dann et al., 2022). Table presents the results our attack on these algorithms, showing that it achieves a 100% ASR across all these algorithms. Additional details about these algorithms provided in Appendix D.
main bandit algorithm. To
8

---

Published as a conference paper at ICLR 2026
ASR Performance vs Hidden Size
ASR Performance under Random Perturbation
1020
D 5920
D 10200
ASA (50
20
Figure 4: Left: Attack success rate as function of the hidden layer width in the reward model (T' 100). The results show that sufficiently wide hidden layer is required for the attack to succeed; once the width exceeds 750 neurons, the ASR reaches 100%. Right: Attack success rate versus the & norm of random noise perturbations. The results show that even with increasing perturbation magnitude, the ASR remains nearly constant, demonstrating that random noise largely ineffective in attacking the bandit algorithm.
4.5 OUR PROPOSED DEFENSE
We propose a defense that remains effective even when environment aware of attack method, has access to the logged data, and is able to shuffle it. The defense operates by randomly shuffling a portion of the logged data before running the bandit algorithm. This simple yet effective strategy substantially reduce the attack's 's success. As shown in Figure 2, shuffling only T/2 of the logged data sufficient to significantly mitigate the attack. Additional results exploring other fractions of shuffled data are provided in Appendix G.
4.6 EFFECT OF RANDOM NOISE PERTURBATIONS ON ATTACK PERFORMANCE
We demonstrate that applying random noise perturbation, produce effective attacks. As shown in the right part of Figure 4, when no perturbation is applied, the attack success rate (ASR) is approximately 25%. Increasing the & norm of the random perturbation has little impact on the ASR, which remains nearly constant across wide range of perturbation magnitudes. This indicates that simple random noise is largely ineffective compromising the bandit algorithm.
with larger l norm, fails to
4.7 EFFECT OF NON-LINEAR REWARD MODEL WIDTH ON ATTACK SUCCESS
In this experiment, we investigate the impact of the reward model's width on attack performance. For our theoretical assumptions to hold practice, the network's linear approximation must be valid. To achieve this, the reward model is designed with single hidden layer of sufficient width. As the number of neurons increases, the linear approximation becomes more accurate (Jacot al., 201 8). Our results indicate that having enough neurons in this hidden layer is critical for successful attack. As illustrated in the left part of Fig. 4, once the number of neurons exceeds 750, the attack success rate reaches 100%. Implementation details of the reward model training for synthetic data are provided Appendix
4.8 COMPARISON OF OUR METHOD WITH ROBUST STOCHASTIC BANDIT ALGORITHMS
Existing corruption-robust MAB works (Niss & Tewari, 2020; Lykouris et al., 2018; Gupta et al., 2019) assume adversary who actively modifies the realized rewards during the bandit interaction. ie., the adversary active, present throughout training, and directly alters the reward feedback as is generated. In contrast,
attack perturbs the internal weights of the reward model before bandit training begins. The adversary does not have access to the environment where the bandit run, which believe is much more realistic setting. However, additionally evaluated

---

Published as a conference paper at ICLR 2026
attack against the Fast- Past-Slow algorithm (Lykouris et al., 2018), representative robust bandit method. As shown Figure 5, this algorithm also vulnerable. With a corruption parameter of 0.5 (a typical setting in their experiments). our attack succeeds 100% of the time, while keeping the perturbation norm small (around 0.3).
Conruation
10
Parturbeic Kod Partubeacc
0.325
68
06
0.275
044
02
Attack En
60
Attack
2.00
0.35
3.50
Figure 5: Our pre-training attack successfully hijacks the Fast flst robust bandit (Lykouris et al., 2018), achieving 100% success with minimal lz-oomm perturbation ( 0.3) for E (3,5), 1000, and d 1000.
Moreover, we evaluate our attack against the e-contamination algorithm (Niss & Tewari, 2020), robust variant of UCB. Figure 6 illustrates that for reasonable contamination level (c 0.15). our attack achieves a 100% ASR while requiring only a small f--norm perturbation (approximately 0.2).
Nerturbation tpaioon
300
0.35
0.23
Noturbeten (Tw200
28
Acturbaten (1w200 Keli Actutottr (143300 Bac)
0.25
Figure 6: Effectiveness of our attack against the e-contamination algorithm (Niss & Tewari, 2020), a robust UCB variant. Experiments are conducted with E {100, 1000}, K E {3,5}, = 1000, 0.1, 1, using the a-trimmed strategy.
5 CONCLUSION
In this work, we investigate the vulnerabilities of offline bandit algorithms used to evaluate machine learning models, including generative models. We show that publicly sharing reward model weights on platforms like Hugging Face exposes these models to adversarial attacks and highlight the crit- ical role of logged data in offline bandit evaluation. We introduce novel threat, demonstrating that even small. imperceptible perturbations to reward model weights can completely hijack bandit behavior. Our analysis starts with linear reward models, revealing their high susceptibility to manip- ulation in nigh-dimensional settings, and extends to nonlinear models such as ReLU-basee neural networks. We validate our approach on real data by downloading publicly available weights from Hugging Face and successfully attacking these models. To mitigate such attacks, we propose de- fense strategy and, for scenarios where the attacker only aims to prevent optimal arm selection, we introduce an efficient on on-off heuristic that is both fast and effective. Finally, provide theoretical support showing how perturbations can systematically steer bandit behavior proportion to input dimensionality.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint Sakiv:2303.087722 2023.
Kingma DP Ba J Adam et al. A method for stochastic optimization. arXiv preprint arXiv: 14X2::14122.900, 1412(6), 2014.
Abien Fred Agarap. Deep learning using rectified linear units arXiv:1803:08335 2018.
(relu).
arXiv preprint
Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of machine learning research, 3(Nov):397-422, 2002.
Olivier Chapelle, Eren Manavoglu, and Romer Rosales. Simple and scalable response prediction for display advertising. ACM Transactions on Intelligent Systems and Technology (TIST), 5(4):1-34, 2014.
Jaemin Cho, Yushi Hu, Jason Baldridge. Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi FosteT and Su Wang. Davidsonian scene graph: Improving reliability in fine- grained evaluation for text-to-image generation. In ICLR, 2024.
Ferdinando Cicalese, Eduardo Laber, Marco Molinaro, al. Teaching with limited information on the learner's behaviour. In International Conference on Machine Learning, pp. 2016-2026
PMLR, 2020.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning. pp. 1310-1320. PMLR, 2019.
Xiangxiang Dai, Yuejin Xie, Maoli Liu, Xuchuang Wang, Zhuohua Li, Huanyu Wang, and John Lui. Multi-agent conversational online learning for adaptive IIm response identification. arXiv preprint arXiv:2501.0188 00189999 2025.
Chris Dann, Yishay Mansour, Mehryar Mohri, Ayush Sekhari, and Karthik Sridharan. Guarantees for epsilon-greedy reinforcement learning with function approximation. In International confer- ence on machine learming, pp. 4666-46688 4689. PMLR, 2022.
Sanjoy Dasgupta, Daniel Hsu, Stefanos Poulis, and Xiaojin Zhu. Teaching black-box learner. In International Conference on Machine Learning, pp. 1547-1555. PMLR, 2019.
Evrard Garcelon, Baptiste Roziere, Laurent Meunier, Jean Tarbouriech, Olivier Teytaud, Alessandro Lazaric, and Matteo Pirotta. Adversarial attacks linear contextual bandits. Advances in Neural Information Processing Systems, 33: 13342-1333333, 2020.
Aurélien Garivier, Tor Lattimore, and Emilie Kaufmaann On explore-then-comme strategies. Ad- vances in Neural Information Processing Systems, 29, 2016.
David M Gay, Michael L Overton, and Margaret H Wright. A primal-dual interior method for nonconvex nonlinear programming. In Advances in Nonlinear Programming: Proceedings of the 96 International Conference Nonlinear Programming, pp. 31-56. Springer, 1998.
Eugene Golikov, Eduard Pokonechnyy, and Vladimir Korviakov. Neural tangent kernel: A survey. arXiv preprint arXiv:2208. 2wtp22208.33611 13144, 2022.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.129453 2025.
Anupam Gupta, Tomer Koren, and Kunal Talwar. Better algorithms for stochastic bandits with adversarial corruptions. In Conference Learning Theory, pp. 1562-1578. PMLR, 2019.
Mohammad Hajiesmaili, Mohammad Sadegh Talebi, John Lui, Wing Shing Wong, et al. Adversar- ial bandits with corruptions: Regret lower bound and no-regret algorithm. Advances in Neural Information Processing Systems, 33:19943-19999 2020.

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 53:6840-6833 2020.
Seyed Mohammad Hadi Hosseini, Amir Mohammad Izadi, Ali Abdollahi, Armin Saghafian, and Mahdieh Soleymani Baghshah. T2i-fineeval: Fine-grained compositional metric for text-to-image evaluation. arXiv preprini 2rXiv:2503. 1481, 2025.
Xiaoyan Hu, Ho-fung Leung, and Farzan Farnia. A multi-armed bandit approach online selection and evaluation of generative models. arXiv preprint arXiv:2406.0744 2024a.
Xiaoyan Hu, Ho-fung Leung, and Farzan Farnia. An online learning approach to prompt-based selection of generative models and llms. In Forty-second International Conference on Machine Learning, 2024b.
Xiaoyan Hu, Ho-Fung Leung, and Farzan Farnia. Pak-ucb contextual bandit: An online learning approach to prompt-aware selection of generative models and 1lms. In Proceedings of the 42nd International Conference on Machine Learning, volume 267, pp. 24447-24481, 2025.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 20406-20417, 2023.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: eralization in neural networks. Advances neural information processing systems, 31, 2018.
Convergence and gen-
Kwang-Sung Jun, Lihong Li, Yuzhe Ma, and Jerry Zhu, Adversarial attacks on stochastic bandits.
Advances in neural information processing systems, 31, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Volodymyr Kuleshov and Doina Precup. Algorithms for multi-armed bandit problems. arXiv preprint 4rXiv:1402.6000 2014.
Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learning to rank in the cascade model. In International conference machine learning, pp. 767-776.
PMLR, 2015.
LAION-AL westhetic-predictor http://ghhhbbconnAAAAoAAAlllliie 2021.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-44; 2015.
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li. Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406. 13743, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision lision-language understanding and generation. In International conference machine learning, pp. 12888-12900. PMLR, 2022.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual- bandit approach to personalized news article recommendation. Proceedings of the 19th international conference on World wide web, pp. 661-670, 2010.
Fang Liu and Ness Shroff. Data poisoning attacks on stochastic bandits. In International Conference on Machine Learning. pp. 4042-4050. PMLR, 2019.
Guanlin Liu and Lifeng Lai. Action-manipulatee attacks on stochastic bandits. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 3112-3116 16. IEEE, 2020.
12

---

Published as a conference paper at ICLR 2026
Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adver- sarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pp. 114-122, 2018.
Yuzhe Ma, Kwang-Sung Jun, Lihong Li, and Xiaojin Zhu. Data poisoning attacks in contextual bandits. In International Conference on Decision and Game Theory for Security, pp. 186-204. Springer, 2018.
Seyed-Mohsen Moosavi- -eeaoil,,, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.
Laura Niss and Ambuj Tewari. What you may not be what you get: Uch bandit algorithms robust to epsilon-contanination. In Conference on Uncertainty in Artificial Intelligence, pp. 450-459.
PMLR, 2020.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PmLR, 2021.
Parham Rezaei, Farzan Farnia, and Cheuk Ting Li. Be more diverse than the most diverse: Optimal mixtures of generative models via mixture-UCB bandit algorithms. In The Thirteenth Interna- tional Conference on Learning Representations, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684- 10695, 2022.
Yuta Saito and Masahiro Nomura. Hyperparameter optimization can even be harmful in off-policy learning and how to deal with it. arXiv preprint arXiv:2404. /5084, 2024.
Jaskirat Singh and Liang Zheng. Divide, evaluate, and refine: Evaluating and improving text-to- image alignment with iterative vqa feedback. Advances in Neural Information Processing Sys- tems, 06.777999-708111 2023.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International conference machine learning, 2014.
Gemini Team, Rohan Anil, Sebastian Borgeaud, can-itatiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1880, 2023.
1805,
Matilde Tullii, Solenne Gaucher, Nadav Merlis, and Vianney Perchet. Improved algorithms for contextual dynamic pricing. Advances in Neural Information Processing Systems, 37:126088- 126117, 2024.
Arkhipkin Vladimir, Viacheslav Vasilev, Andrei Filatov, Igor Pavlov, Julia Agafonova, Nikolai Gerasimenko, Anna Averchenkova, Evelina Mironova, Bukashkin Anton, Konstantin Kulikov, et al. Kandinsky 3: Text-to-image synthesis for multifunctional generative framework. In Pro- ceedings of the 2024 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 485, 2024.
Martin Wainwright. High- digh--dimessooall statistics: A non-asymptotic viewpoint, volume 48. Cam- bridge university press, 2019.
Jiazheng Xu, Xiao Liu, Yuchen WL, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 1595555 2023.
13

---

Published as a conference paper at ICLR 2026
A RELATED WORK
While adversarial attacks on supervised learning models have been extensively studied (Szegedy ct al., 2014; Moosavi-Dezfooli ct al., 2017: Cicalese et al., 2020; Cohen et al., 2019; Dasgupta et al., 2019), the vulnerabilities of bandit algorithms remain less explored. This is particularly important in settings where reward models (Singh & Zheng. 2023; Cho et al., 2024; Hu et al., 2023; Hos- seini et al., 2025) are combined with bandit algorithms evaluators (Rezaei al., 2025; Hu al., 2024b), highlighting the need to understand prior adversarial attacks on bandit algorithms and to prepare for developing new attack strategies. Recent studies have investigated adversarial attacks on multi-armed bandits (MABs) (Jun et al., 2018; Liu & Shroff, 2019; Liu & Lai, 2020) and linear con- textual bandits (Ma et al., 2018; Garcelon et al., 2020). In reward poisoning attacks, an adversary manipulates the reward signals received by the agent from the environment (Jun et al., 2018; Liu & Shroff, 2019; Niss & Tewari, 2020; Lykouris et al., 2018; Gupta et al., 2019; Hajiesmaili et al., 2020). By subtly altering these rewards, the attacker can bias the learning process, causing the agent to favor specific actions or arms, which may lead to suboptimal or adversary- drivven behavior, In lin- ear contextual bandits, adversarial attacks have focused on two main strategies. The first is reward poisoning, where the adversary modifies the rewards associated with chosen actions to steer the agent toward particular arms (Ma et al., 2018; Garcelon et al., 2020). The second is context poison- ing, in which the attacker alters the observed context vectors while leaving the rewards unchanged, attacker-desired decisions (Garcelon et al., 2020).
misleading the agent into making suboptimal Together, these attacks demonstrate the vulnerabilities of linear contextual bandits to both reward and context-based manipulations. Our work differs from prior studies that, rather than directly at- lacking the rewards, we target the reward models themselves. This approach is particularly relevant in the emerging usage of bandit algorithms as evaluation metries. We show that in high-dimensional settings, reward models are vulnerable, making it possible for an attacker to easily manipulate them and thereby hijack the behavior of the bandit algorithm.
B PROOFS
Proof of Theorem 3.1. We first establish the result for the case of full-traeectory attacks. The other two variants differ only in that they employ a subset of the (T' K) )K( 1) constraints: hence, both the optimization formulation and the feasibility properties can be deduced directly from the analysis of the full-trajectory case. In this regard, define the residual outputs Z as Z,(1 x p
therefore we have ] for all valid and j. Let us define s Rd for i E [K] and k [m.]] as
μs +
In this regard, we have E| sub-problem above reduces to
H. for all i and regardless of l 2 1. Therefore, the second
min | l|2
subject to (w +)
( (
s(1) k2
> 22g (1) -1)-);))+ (N, 21)
(for i == Ār, Vj +i, K <(ST),
and N.(t) - E ((, = i). The above set of inequality constraints can be further simplified via introducing new symbols, and hence the above convex feasibility problem can be rewritten as
mig |
(8)
subject to 5 Ty R3,,, ej [K -1, 1), K <tST,
14

---

Published as a conference paper at ICLR 2026
where the vectors T and scalars R3 depend samples in datasets Dj and the target sequence Ar. More precisely, we have
N.(1)
33 (H,: -1p) + N((t)
Σ 2
1
Σ
z
(9)
(y((t)
i=1
i=1
R3,0 w T + v logt N 112 N₁ 1/2 (t))
where we have i _ A, and j' E K 1] is the index of the set [K] after removing A.
In the where the reward model is neural network rather than linear, the NTK approximation in equation 5, together with the assumption max; W; > 1, yields an analogous QP form, with modified definitions of T5 and Rj.t:
N.(1)
Ny
T
VoNNo(X N.
E VONNo(X("))
(10)
(.(
E
N.(0)
Ng((1)
Ryjt A
Σ
NNo(X
Ny(()) NNo(X(73)
+ 22log (N-190)-N7110
(1)
This completes the proof.
Proof of Theorem 3.3. Without loss of generality, we assume i" Recalling the notation from the proof of Theorem 3.1, for all valid j and f, we have
A, A 111111
E W T5
(11)
A 1,j=1
1
Var [ Ty N:(1)
w CoVp, w Nyy(1)
w Covp, W.
Note that both T3 and R are deterministic. Also, we have
R S w TwTTe Tyal + 22 (1 (1-t-1/2
(12)
The set of constraints can be rewritten in the following matrix form:
T - R.
(13)
where d matrix T and C-dimensional vector are formed via row-wise concatenation of j and R,.s, respectively. Here, C denotes the number of constraints which is at most (K (K-1)(T-K)) Note that we have replaced with > by assuming infinitesimally small margins in the constraints. This modification simplifies the subsequent part of the proof. Also, it should be noted that both T and are random.
We need to show that the stochastic inequality system T R is feasible with probability 1. We proceed by contradiction. Without loss of generality, assume that the first inequality T. > R infeasible (with positive probability), given that the remaining C C-1 inequalities are satisfied. Formally, assume
(((0)
I 8 (
Ra n { 1
TT 82 Ri i= ..... cJ =
2) >0
Then we must have
P (( 5 | T = R) = 00>000
that is, the stochastic equation system T5 R is unsolvable with positive probability. Since the number of variables d exceeds the maximum number of equations (T K)(K 1), this only occur if the rows of fail to be linearly independent with positive probability.
However, this is impossible by construction of T in equation 9. Indeed, since all resid- uals (2(5) Jjein]] are independently drawn and each distribution P; is non-degenerate (i.e., Amin (P)) > 0 for every [K]), the rows of are distributed general position and are therefore linearly independent with probability 1. This completes the proof.
15

---

Published as a conference paper at ICLR 2026
Proof of Theorem 3.4. We begin with the following lemma:
Lemma B.1. Assume (T KK 1) < and consider the constrained optimization problem
8 _ arg min || | ll subject to T R.
(14)
Define δ A T where i denotes the Moore-Penrose psendo-inverse, i.e., ő (TT ) -IT R. Then we have || 8 1|a 1
Proof. The argument is straightforward. Removing any constraint from equation 14 (equivalently, removing row of T cannot increase || ö* ||2. Hence, the maximum possible fo-norm of 8 occurs when all constraints are active. Formally,
|| 8 1|2
arg min 8 subject to
T = R
(15) where the last equality follows from the definition of Moore-Penrose pseudo-inverse, which yields the unique minimum-f2-norm solution to an underdetermined linear system. Also, according to Theorem 3.3, know the linear equation system T R is solvable (hence 5 exists) with probability 1. This completes the proof.
|
Next, we bound || In order to do this, first note that
|
(16)
min (T)
where minn (-) denotes the minimum singular value. Therefore, we need to: i) find a high-probability bound for miin (T)) and ii) establish similar bound for |||||2 This requires an appropriate matrix representation for the construction of T, whose rows are defined in equation 9:
(C(C)
(1 Σ
any row of T p4; paj
Σ
Z 1
2
(17)
N.C
E=1
1=1
for any two properly defined indices j.j. We then leverage existing tools from random matrix theory to establish our main results.
Matrix-form generation of T. Let define the residual matrix I as follows:
T
(1)
z!!! (1)
K
Z
(K) [
(18)
which is an (n1 + +... + ng) x d dimensional matrix that collects all the Z vectors as its rows. According to the assumptions in the statement of the theorem, i' random matrix with independent rows. Moreover, since each data-generating distribution is assumed to be a product measure, the entries in each row are also independent and have variance 1. We are particularly interested
sub- matrix of I, denoted by I* E ((T-K)xd which consists only of those vectors Z ()) that correspond arms pulled during the target trajectory for time-steps K < T. Since the target trajectory fixed and deterministic, I* is still a random matrix with independently distributed entries of unit variance.
Recall that (T) denotes the number of times arm has been pulled up to the horizon T for a given fixed target trajectory. For any non non-degenerate trajectory A,, we have N₂(7) < O(T/K), vi e [K]. Let m denote the maximum number of times any arm has appeared during the trajectory, so that OTT/K). Now, consider the following family of matrices:
1/2
1/n
0
1/2
1/n
A
Vn E N.
(19)
0
0
1/n
16

---

Published as a conference paper at ICLR 2026
Lemma B.2. have Amin (An) 1/n.
Proof. Proof is straightforward. For some fixed n E N. let us denote the eigenvalues of An by A1,...,AAp An. Then, have det(An, X;I) for all [n]. On the other hand, since A, A₂ is upper-triangular, we have
det(An - X1 II 7 - -1), Vj e In].
Therefore, we must have Xj = 1/j for all j E [r]] and consequently Amin (An) 1/n, which completes the proof.
L
Consider the matrix Am. and let us denote its columns by Ui..... Um Rm. Based on the definition Tf T in equation 9, each row of consists of mean-difference term μ4 - μ4 (for properly defined indices i, j), together with a term that depends solely residuals. This residual part can be rewritten
as
N.(O
N₂ Σ z = (10 UN.()) | - [ | 1o] r",
Σ
(20)
NO(1)
N₂(1
where [0] vs 0] denotes vector of dimension T - K obtained by padding v; with zeros both before and after. this regard, the random matrix can expressed as
T= T= + VTT"
(21)
where M a low-rank matrix consisting of the mean-difference terms f4; - μ4 as rows. This includes (potentially) all possible pairwise differences between the means of the K distributions Pi...., Pk. with repetitions as required. Note that M can have at most K nonzero singular values. collects all combinations of vectors of the form
The matrix V ((T-K)x(T-
o UN.(O 0] |o
UN,(0 [0]
its columns. The matrix V is fully determined by the target trajectory and can be regarded row/column-wir wise augmentation Am- Therefore,
min (V) ((mmin(Am)) 0(1mm) 2 O(K/T).
(22)
Consequently, obtain
minn (T) 2 O(K/T) Gmin (T*).
(23)
What remains, with respect to bounding the denominator in equation 16. is to establish a high- probability bound mmin (L. To this aim, invoke well-established result from random matrix theory. There are several known bigh-probability bounds for the minimum and maximum singular values of random matrices with independently drawn entries of equal variance have:
particular, we
Theorem B.3 (Based on Theorem 6.1 of Wainwright (2019)). For any positive 5, and assuming d > T K, we have
Tiiu (T*) 2 vd VT K,
(24)
with probability at least 1 - -66
The proof can be found in the reference. Therefore, for any >
Kvd T
IP Mmin (T) o
(1-6 -00--22 T==)) > 10--061/2
-d2-/2
(25)
Here, used the fact that Mata-generating distributions product measures with independent entries, such as N Nuu, I²I) for any mean vector and variance-per per-imesprri o². Next, we try to (with high probability) upper- the nominator of equation 16.
17

---

Published as a conference paper at ICLR 2026
Number of Constraints in OSA Compared with log(T)
20
K-3
18
log(T)
16
14
Constraint
12
10
4-
2-
2000D
40000
5000D
80000
100000
Time Horizon (T)
Figure 7: Number of constraints in the OSA method as function of the time horizon T. The results show that the number of constraints grows logarithmically with T (O(log T)) for both K = 3 and K arms, explaining the high efficiency of the method.
High-probability bounds on |||| What remains is to bound | RRIl2 in equation 16. From equa- tion 11 and equation 12, and using Chernoff bound (Wainwright, 2019), we obtain naive high- probability bound as follows:
PP P( 2 0112 + VlogT) < e
Ve> 0, i [[K 1)(T K)]-
(26)
Therefore, using (7 - K)(K - 1) 1) < KT and assuming T >> 1, we have
Rll2 2 O (VcKTlogT) < vc>0.
(27)
Combining equation 25 and equation 27, and applying the union bound, we obtain the following high-probability bound for ||0|/2 (and hence (12), for any d, c 0:
p |101| s o 9/2 1/o///lo'T
Ve
2 -
(28)
VKd
1- VT Ky/d
Finally, choosing log and 5 = d 1/3 yields the bound stated in the theorem and completes the proof.
C ANALYSIS OF CONSTRAINT NUMBER IN THE OSA METHOD
We study the effect of increasing the time horizon T the number of constraints in the OSA method. Our observations show that, while the number of constraints grows with T, this growth follows an Olog T') pattern, as illustrated in Fig. for ranging from to 100,000. To confirm this behavior, repeat the experiment for two values of the number of arms, K = 3 and K 5, and observe the same pattern in both cases. This small number of constraints is one of the main reasons that makes the OSA method so fast.
D ADDITIONAL DETAILS ON ETC AND E-GREEDY ALGORTHTS
D. ETC ALGORITHM
The Explore-Then-Comee Then-Commit (ETC) (Garivier et al., 2016) algorithm a simple bandit strategy that divides the learning process into two distinct phases. During the exploration phase, the algorithm
18

---

Published as a conference paper at ICLR 2026
Table I: Comparison of attack performance on linear and non-linear reward models using the e- greedy and ETC methods. All attacks achieve a 100% success rate (ASR 100). The table illustrates how varying the parameters K and (or max; Wi) affects the attack duration. In this experiment, we set T 100 and the number of exploration rounds for ETC to m 5.
Reward Models
Linear Reward Model
Non-linear Reward Model
K
d
Attack Time
K
max; W;
Attack Time
c-greedy Attack
10000
00:00
3
3000
00:22
1000
00:00
3
6000
00:12
10000
00:00
5
3000
00:18
1000
00:00
5
6000
00:13
Explore- Thenee---en-Caaee Chee-C (ETC)
10000
00:00
3
3000
00:03
1000
00:00
6000
00:06
5
10000
00:00
5
3000
00:04
1000
00:00
6000
00:09
uniformly samples each arm predetermined number of times gather initial reward estimates. Once the exploration phase is complete, ETC commits to the arm with the highest estimated reward for the remainder of the time horizon. This approach provides balance between exploration and exploitation while remaining easy to implement and analyze. The length of the exploration phase can be tuned to optimize performance depending on the problem setting.
To attack ETC, the adversary only needs to wait until the exploration phase is complete. At the transition between the exploration and commit phases, the adversary manipulates the rewards so that the empirical mean of the target arm exceeds that of the optimal arm. During the subsequent commit phase, the algorithm then selects the target arm. Since the ETC constraint set involves only single constraint, the attack is fast and straightforward, particularly in high-dimensional settings.
D.2 G-RREDYY ALGORITHM
The e-greedy (Dann et al., 2022) algorithm selects random arm with probability e (exploration) and the arm with the highest estimated reward with probability 1-e (exploitation) each time step. In our experiments, we use an adaptive that decreases over time, allowing the algorithm to explore more during the early stages and gradually exploit the best-known arm as more information is gath- ered. This time-dependent strategy enables the algorithm to balance exploration and exploitation effectively, and the choice of the initial e and its decay schedule directly influences its performance.
To attack e-greedy, the adversary follows the same principle OSA. During random selections, no attack is performed. However, when the algorithm selects the arm with the highest empirical mean, the adversary intervenes to prevent the optimal arm from being pulled. Specifically, the target arm chosen as the runner-up, and its empirical value is increased to surpass the optimal arm, as modifying the runner-up is simpler than other arms. This approach makes the attack fast and highly efficient.
For the ETC algorithm, we set the number of exploration rounds selected exactly five times before the commit phase begins. For the e-greedy algorithm, we set the initial to 0.1 and apply decay over time, with a minimum value of 0.01.
5, meaning each arm is
E IMPLEMENTATION DETAILS OF REWARD MODEL TRAINING FOR SYNTHETIC DATA
We train several neural networks, each with single hidden layer whose size varies depending the task. The networks are optimized using Adam (Adam et al., 2014) and employ the ReLU activation function (Agarap, 2018). Training is performed using the mean squared error (MSE) loss. The
19

---

Published as a conference paper at ICLR 2026
10
540AL <Sa
sthe Pearca Lilket
Facculyrte Uiben
beattrey Libert
(a) ASR before and after defense the linear reward model when T/4 logged data shuffled. defense significantly reduces the attack success rates across all attack methods.
(b) ASR before and after defense the linear reward model when 3T/4 /4 the logged data shuffled. The defense further reduces attack success rates compared to the T / case.
Figure 8: Comparison of ASR before and after defense on the linear reward model under different fractions of logged data shuffled as a defense.
logged data split into training and validation sets, with 80% used for training and 20% reserved for validation. Networks are trained for 100 epochs with a batch size of 1024 and a learning rate of 10-3, For supervision, the ground truth computed the inner product of the data with the mean of the optimal arm.
F ADDITIONAL RESULTS ON ATTACKING THE REAL DATASET
To demonstrate the generality of our attack, in addition to two pretrained reward models com- monly used to evaluate generative image models, Image Reward (Xu al., 2023) and Aesthetic Model (LAION-AI, 2021), we attack randomized reward network. This random reward model initialized with random weights and we freeze the majority of its parameters except for a single hidden layer. Despite its random initialization, the model's trajectory can still be hijacked by our method: the ASR distribution for this random reward model closely matches those of the Image Re- ward and Aesthetic models in Figure 3. Figure 9 shows the distribution of our attack the random reward model.
Distribution of ASR values for Random Reward Model
Tw100, K-3
0.9
T-100,
0.8
Probabiity
0.3
50
10
90
Figure 9: ASR distribution on the random reward model, showing vulnerability similar to the Image Reward and Aesthetic models despite random weights.
G ADDITIONAL RESULTS ON THE DEFENSE METHOD
The effectiveness of the shuffling defense can be further evaluated by varying the fraction of logged data that shuffled. In particular, we consider two additional settings where T/2 and 3T/4 of the logged data are shuffled before running the bandit algorithm. The results demonstrate that increasing
20

---

Published as a conference paper at ICLR 2026
the portion of shuffled data substantially reduces the attack success rate (ASR) across all attack methods. Detailed results for these settings are presented in Figure 8.
H GEOMETRY OF ATTACK VECTOR
We visualize the geometry of the attack vector relative other vectors, reduce the perturbation space to three dimensions using PCA. As shown in Fig. 10a and Fig. 10b, the perturbations are neither trivial nor simply the reverse of the vector w. This demonstrates that the attack is subtle, imperceptible, and not easily detectable from the bandit algorithm's 's perspective.
impect of Vectors
Impact
14
14
1.0
05
655
100
13.4 03
0.0
1P.1.
14 -25
02
08
55
10
-1.
-1.
(a) T = 100, K 3, D = 1000, OSA
(b) T' 100, K = 4, D = 1000, OSA
Figure 10: 3D visualization of perturbations for different values of K.
I A REALISTIC AND PRACTICAL THREAT MODEL FOR MODERN BANDIT SYSTEMS
Our setting, similar to most modern and practical MAB formulations, assumes that pulling an arm produces a data sample, not a direct reward value, and the bandit/learner then evaluates the reward based on that sample. For example, an image generator or an LLM returns an image or text when queried (equivalent to pulling an arm), and the learner assesses a reward (e.g., fidelity or quality) using some reward function. Such reward functions may be linear or neural networks. Here, sample generation is handled by the environment, to which we assume no access, whereas reward functions are typically derived from publicly available pretrained models that can be poisoned and later down- loaded by unknowing learner for bandit training. Several of the cited works (Rezaei et al., 2025; Hu et al., 2024b) in our Introduction highlight exactly this kind setting. Another point to mention is that believe attack perturbation requires the adversary to be online, active, and present inside the private server where the bandit is trained. This is actually too much to assume from the adversary's side. We (both theoretically and experimentally) show that poisoning the reward function before training even be- gins can still cause equally significant damage, while the attack itself is much simpler to perform. We believe it is important for the community to recognize that such vulnerability exists in offline bandits, which may motivate future efforts toward developing defense mechanisms.
setting is more realistic in practice than previous attack strategies. Instantiated
J VISUALIZATION OF OSA METHOD ON REAL REWARD MODEL
Figure 11 illustrates our OSA attack on the Aesthetic Reward Model (LAION-AI, 2021). The first panel presents the baseline experiment: UCB (Auer, 2002) is run using the reward scores from the model, which leads it to consistently select SD3 (Rombach et al., 2022) the optimal arm since it achieves the highest aesthetic scores. The plot depicts the arm-selection trajectory over time under this setting. The second panel shows the same experiment under attack. By manipulating the weights of reward model, the attacker hijacks UCB so that it selects the Openjourney model as the optimal arm. Timesteps targeted by OSA method highlighted in gray.
21

---

Published as a conference paper at ICLR 2026
Proege bess parch branch, the nappy are choping boddy the intering slinnty
201
Operpericy
SUDCL
(a) Results with UCB on real data.
Prorge:: Thoo bincs parch anaa brench, happy are chippig koutly, one holecing shenty.
14
(b) Results with OSA on real data.
Figure 11: Comparison of methods on real data. (a) UCB results. (b) OSA results.
22

---

Published as a conference paper at ICLR 2026
K ALGORITHMS
The detailed algorithms for the Full Trajectory Attack (Algorithm 1), OSA Anack (Algorithm 2), and Trajectory-Free Altack (Algorithm 3) are provided below, Each follows the same structure with and update, with differences noted in three functions: select_at find.perturber the comments.
Algorithm 1: Full Trajectory Attack
Function selectuarm (4, K, Ā)
if < K then
return t, False
else
// Adversary selects according to the target trajectory
orm + As return arm, True
Function Find.perturbation (K, S, G. C, {4) use :
// Consider all the inequalities
for j < to K, j & arm do
if use NN then
dy + G G6)
- Plog N,
22og Narm
+ (F())
p(arm))
else
dj f- g(arm)
-86)
Cj
2log!
N₂
V
2log Narm
11. dj
Append (yj,cy)too Co
M t Ø foreach (d, do
L M - M 18T c+ 10
+ solve QP in Eq. equation with constraints M return d
Function update (X, s(arm) p(arm), G(arm), use NN))
Narm Nari +1
if NN then
((rrm)
G ((rrm)
VNN G(arm))
Narm
f(arm)
t p(arm)
(X) - p(arm))
Narm
else
S (arm)
S ((rrm)
((erm))
Narm
// Inputs: (D:) Logged data, Á is target trajectory, (u are inferred means, NS is boolean flag, N.,, the reward model
Input: K, d, (D:), ; use NN, ( Output: Perturbation 8
/| N: counts, S: sample mean, G: mean gradient, F: mean network output, C: constraint set
Initialize N, S, G, F +
Initialize C +- Ø, 8 0
for t < to do
arm, do_attack + select.arm (t. K, A)
ddo.attack then
L 8 find.perturbation (K, S, F. {M]K NN)
Pull arm, observe X ~ Darm
update (X, g(arm), p(arm) ((arm), use NN)
return 8
23

---

Published as a conference paper at ICLR 2026
Algorithm 2: OSA Attack: Online Score-Aware Attack
Function select arm (t, K, :
if é K then
return 1, False
else
// Select the runner-up arm for attack
for j t to K do
UCB + R()) + N;
2og
urm <- arg max, UCB[j] II Attack if the with the highest UCB is the optimal arm do_attack + (arm == 1)
return arm, do attock
Function find.perturbation (S, G, F, C, μt, uSe_NN:
// Construct inequality constraints to prevent selection of optimal arm use_NN then
d ((arm) G(1)
(V2 2logt N1
2 logt Narm
+ (F(1)) p(arm))
ct
else
d + g(arm)
-S(2)
C- (v 2logt
V
Blog Norm
N₂
Append (d, to
M
foreach (d. do
[ M - M U 2
solve QP Eq. equation with constraints .
return 6
Function update (X, glarm), F(arm), G(arm), uNe_NN, 8)
Narm &- Narm +1
use NN then G(arm)
((arm)
(NNN(()) G(arm))
Nurm
F Flamm F (arm)
(X) p(arm))
Narm
else
((arm)
<- ((arm)
(X - S(arm))
N Narm
I/ Update empirical rewards using current 5 Update Rt using
/ Inputs: [D:) Logged data, {u]K inferred means, use.Ny .se.NNia is boolean flag, N.1, the
reward model
Input: K, d, T, (D.) use NN, NNo ( are inferred means Output: Perturbation 0
/ N: counts, S: sample mean, G: mean gradient, F: mean network output, R: empirical rewards, C: constraint
Initialize N,S, G. C.FF+O 0
Initialize < 2,
for t + to do
arm, do_dddaattta atlack +- select. (1, K,N,R) if do_allack then
L 6 + findaperturbation (S, G. F, C, μn, use NN)
Pull arm, observe X ~ Darm
update (X, R, slarm) p(arm) ((arm) use (se_NN,8
return 6
24

---

Published as a conference paper at ICLR 2026
Algorithm 3: Trajectory-Pree Attack
Function
sel sclect arm (1, K)
if K then
return 1, False
else
I/ Select sub-optimal arm round-robin order arm & next sub-optimal arm
return arm, True
Function Eind.perturbation ion (S, G,F, C, P1, use NNN):
// Construct constraints only w.r.t optimal arm use_NN then
d + (arm)
G(1)
2logt ct (V
2log Narm
+F((1) - p(arm))
N1
else
d t g(arm)
-S(2)
ct (v 2logt
V
2logt
u.d
Ni
Narm
Append (d, c) to
M +
foreach (d, c) E do
L M - M U (8T + 10
solve QP in Eq. equation with constraints M return 6
Function update (X, slarm) plarm), G(arm) use NN)
Narm Narm +1
Juse_N then
((arm) t (arm)
VNN - G(arm))
p(arm) p(arm)
(X) p(arm)
else
S(arm)
((arm) Nurm
X S((rm))
// Inputs: (D.) Logged data, fur 1K are inferred means, NN is a boolean flag, No the reward model
Input: K, d, T, (D. J.K {14 J.K use NN
Output: Perturbation 0
I| N:
counts, S: sample mean, G: gradient, F: mean network output, C: constraints
Initialize N, G, +
Initialize C - 0
for 1 - 1 1o 7' do arm, do do_ottack select.arm (t, K) if do attack then
L find perturbation (S, G, F1, use NN)
Pull arm, observe X ~ Darm
update (X, ((arm) plarm). ((arm) use NN)
return 5
25

---

Published as a conference paper at ICLR 2026
L THE USE OF LARGE LANGUAGE MODELS (LLMs)
We used Large Language Models (LLMs) only in a limited way, specifically for minor writing polish and phrasing suggestions.
26


Reviews:
[{"review_id": "3UMmHEdgKg", "reviewer": "Reviewer_sGeV", "summary": "The paper studies reward poisoning for stochastic multi-armed bandits under two reward function classes. It proposes three attack strategies and derives bounds on the maximum per-round corruption required to force errors against upper-confidence-bound style algorithms. The empirical section illustrates the practical impact of these attacks.", "questions": "1. How does the proposed poisoning perform against robust variants of UCB such as [1]?\n2. Theorem 3.4 appears to imply that as $d \\rightarrow \\infty$, the amount corruption needed goes to zero which is contradictory as at least $\\Delta_{min}$(reward gap between best and second best arm) per round corruption is needed to induce suboptimal play for $T \\ge \\frac{K}{\\Delta_{min}^2}$. Could you elaborate on how $d$, $\\Delta_{min}$, and $T$ are scaled in the theorem to avoid this apparent discrepancy?\n\n[1] Niss, L. &amp; Tewari, A.. (2020).  What You See May Not Be What You Get: UCB Bandit Algorithms Robust to $\\varepsilon$-Contamination. Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. The problem setting is stated clearly, and the three corruption procedures are precisely specified.", "weaknesses": "1. The related work on corruption robust multi-armed bandits is not discussed at all, only a few mentioned in Appendix. I suggest the authors to include important works such as [1, 2, 3] and directly compare them by total amount of corruption ( $\\mathbb{E}[ \\sum_t  \\delta^\\top X_t] $).\n\n2. The proposed algorithms is mostly examined against vanilla UCB, and briefly on greedy algorithms, both of which are not designed to be adversarially robust. What would be the minimum $\\delta$ required against such robust MAB algorithms (e.g. [1, 2])?\n\n3. Figure 1 is unclear. The x-axis uses “attack time (seconds)” while the rest of the paper uses discrete rounds. Either convert everything to rounds or explain the mapping between wall-clock time and rounds.\n\n[1] Lykouris, T., Mirrokni, V. and Paes Leme, R., 2018. \"Stochastic bandits robust to adversarial corruptions\". In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing.\n[2] Gupta, A., Koren, T. and Talwar, K., 2019. \"Better algorithms for stochastic bandits with adversarial corruptions\". In Conference on Learning Theory.\n[3] Hajiesmaili, M., Talebi, M.S., Lui, J. and Wong, W.S., 2020. \"Adversarial bandits with corruptions: Regret lower bound and no-regret algorithm\". Advances in Neural Information Processing Systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lY5g4Q3hCp", "reviewer": "Reviewer_8Gpp", "summary": "This paper studied offline adversarial attacks against linear UCB algorithm in high-dimensional space. The attacker has knowledge of the offline dataset associated with each arm, and has the ability to perturb the reward function. There are two scenarios considered in this paper - linear reward function and neural network-based reward. The paper studied 3 different attack goals. Theoretical results are derived to show attack feasibility. Interestingly, the authors showed that when the problem dimension is higher, the attack is easier to achieve. Experiments are done to demonstrate the effectiveness of the proposed attacks.", "questions": "Please help clarify my 3 concerns in the weakness part.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper extended traditional attacks on multi-armed bandits to high-dimensional space, and studied the problem from a different perspective. Instead of focusing on analyzing the attack cost, this paper targets analyzing the behavior of attack as the problem dimension grows. This is a new angle and may bring interesting topics to the community.\n\nBoth theoretical analysis and empirical study are performed the analyze the behavior of the proposed attacks, as the dimension of the bandit grows. An interesting observation is that the attack is easier as the dimension increases.", "weaknesses": "The problem setup is hard to justify in the following sense.\n\n1. The attacker can perturb the reward function. This is too strong power. Traditional attacks only require attackers to perturb instantiated rewards, rather than the underlying reward mechanism. This is saying the attacker need to be able to change the underlying environment completely, which is too demanding.\n\n2. Even if the bandit algorithm is forced to follow certain behaviors under attack, the data observed in each time step is still drawn from an pre-specified offline dataset. This is very weird setup, because attacker should have changed the learner's behavior, and the data should no longer be clean and fresh. This is a critical caveat in the problem setup.\n\n3. The attack feasibility is hard to satisfy. The dimension must upper bound KT, which is a very strict constraint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KONF8yDMzX", "reviewer": "Reviewer_HY31", "summary": "This work represents a breakthrough contribution to the field of adversarial attacks on bandit algorithms, with profound theoretical, methodological, and practical significance. By systematically revealing the extreme vulnerability of offline multi-armed bandit algorithms to reward model perturbations in high-dimensional scenarios, the authors construct a rigorous and cohesive research framework integrating theory, algorithms, and experiments. The novel threat model, clear theoretical characterization of high-dimensional vulnerability, efficient attack strategies, and comprehensive validation on real-world reward models address a critical gap in the community—especially amid the growing reliance of generative model evaluation on bandit methods. However, the work faces non-trivial limitations in threat model assumptions, defense robustness, experimental depth, theoretical generalization, and ethical discussion.", "questions": "Extend the analysis to \"partial information\" (e.g., attacker has access to a subset of offline data) and \"black-box reward model\" scenarios, discussing the feasibility of attacks under these more realistic settings. Conduct sensitivity analysis on data sampling errors and model parameter uncertainty, quantifying how deviations from the ideal assumption affect attack performance. Relax the idealized assumptions in Theorem 3.4, extending the theoretical results to non-identity covariance matrices or correlated features.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Offline bandits have been widely adopted in generative model evaluation, but existing adversarial research mostly focuses on online settings or direct reward tampering. This paper pioneers the systematic study of a new threat model—\"attackers only perturb pre-trained reward model weights\"—filling a critical gap in the field. The problem definition is highly relevant to real-world evaluation scenarios, providing valuable insights for the community.", "weaknesses": "The paper assumes the attacker has full access to the same offline dataset as the victim, can modify the reward model weights before the victim trains the bandit, and the victim will use the tampered weights for evaluation. In practical evaluation scenarios, data and hyperparameters are often not fully disclosed to attackers, which limits the work’s practical applicability. The proposed data shuffling defense is only effective under the ideal setting where \"the attacker fully knows the original sample order.\" If the attacker constructs perturbations based on statistical properties rather than sample order, the defense may fail. Additionally, there is no theoretical quantification of the utility-robustness tradeoff for the defense. Theorem 3.4 assumes the data distribution is a product measure with identity covariance matrix, and the mean of the optimal arm is orthogonal to that of the second-best arm. This idealized setting may not hold for high-dimensional natural images, limiting the theoretical results’ generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)