Title: Adversarial Attacks on Online Learning to Rank with Click Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: online learning to rank

adversarial attack

click model
Abstract: Online learning to rank (OLTR) is a sequential decision-making problem where a learning agent selects an ordered list of items and receives feedback through user clicks. Although potential attacks against OLTR algorithms may cause serious losses in real-world applications, there is limited knowledge about adversarial attacks on OLTR. This paper studies attack strategies against multiple variants of OLTR. Our first result provides an attack strategy against the UCB algorithm on classical stochastic bandits with binary feedback, which solves the key issues caused by bounded and discrete feedback that previous works cannot handle. Building on this result, we design attack algorithms against UCB-based OLTR algorithms in position-based and cascade models. Finally, we propose a general attack strategy against any algorithm under the general click model. Each attack algorithm manipulates the learning agent into choosing the target attack item $T-o(T)$ times, incurring a cumulative cost of $o(T)$. Experiments on synthetic and real data further validate the effectiveness of our proposed attack algorithms.

Adversarial Attacks on Online Learning to Rank with Click Feedback

Abstract

Online learning to rank (OLTR) is a sequential decision-making problem where a learning agent selects an ordered list of items and receives feedback through user clicks. Although potential attacks against OLTR algorithms may cause serious losses in real-world applications, there is limited knowledge about adversarial attacks on OLTR. This paper studies attack strategies against multiple variants of OLTR. Our first result provides an attack strategy against the UCB algorithm on classical stochastic bandits with binary feedback, which solves the key issues caused by bounded and discrete feedback that previous works cannot handle. Building on this result, we design attack algorithms against UCB-based OLTR algorithms in position-based and cascade models. Finally, we propose a general attack strategy against any algorithm under the general click model. Each attack algorithm manipulates the learning agent into choosing the target attack item T -o ( T ) times, incurring a cumulative cost of o ( T ) . Experiments on synthetic and real data further validate the effectiveness of our proposed attack algorithms.

1 Introduction

Online learning to rank (OLTR) has been extensively studied [1, 2, 3, 4, 5] as a sequential decisionmaking problem where, in each round, a learning agent presents a list of items to users and receives implicit feedback from user interactions. One of the most common forms of feedback considered in literature is in the form of user clicks [1, 4, 5]. OLTR with such click feedback can lead to major improvements over traditional supervised learning to rank methods [6, 7, 8]. However, there is a security concern that user-generated click feedback might be generated by malicious users with the goal of manipulating the learning agent. Understanding the vulnerability of OLTR under adversarial attacks plays an essential role in developing effective defense mechanisms for trustworthy OLTR.

There has been a surge of interest in adversarial attacks on multi-armed bandits [9, 10, 11]. For example, [9] shows that, for stochastic bandits, it is possible to manipulate the bandit algorithm into pulling a target arm very often with sublinear cumulative cost. Though it generally follows the bandit formulation, it differs from stochastic bandits in the action space and feedback model. More specifically, OLTR chooses a list of K out of L arms, instead of just one arm, to play in each round; the realized rewards of the chosen arms are usually censored by a click model, e.g., position-based [4] or cascade model [5]. Thus, it is nontrivial to design efficient adversarial attacks on the censored feedback of the chosen arms for OLTR.

Moreover, previous works [9, 10] can only handle unbounded and continuous reward feedback, while the (binary) click feedback of OLTR is bounded and discrete. Such binary feedback brings new

† ∆ 0 : parameter of the attack algorithm; ∆ a : mean gap between arm a and L ; β : a decreasing function in Section 3.2

challenges to the attack design against OLTR algorithms. Since the post-attack feedback must also be binary, this scenario introduces a new problem of deciding whether to attack when the output attack value from previous algorithms is between 0 and 1 : a simple rounding up might be costly, while skipping the attack may lead to undesired non-target arm pulls. Furthermore, the attack value computed by previous attack algorithms can be larger than 1 , which is higher than the maximum attack value in the click model. In other words, in the bounded and discrete model, it is impossible to always find a feasible attack value to ensure the required conditions for their theoretical guarantees.

Contributions. In this paper, we propose the first study of adversarial attacks on OLTR with click feedback, aiming to overcome the new issues raised by the OLTR click model as well as the binary feedback. Table 1 summarizes our proposed attack algorithms with their theoretical guarantees. Since the click feedback itself complicates the attack design, we first consider adversarial attacks on stochastic L -armed bandits with Bernoulli rewards where the feedback from the chosen arm is always binary. We propose an attack algorithm that can mislead the well-known UCB algorithm [12] to pull the target arm L for N L ( T ) times in T rounds, with a cumulative cost C ( T ) in the order of log T asymptotically. Based on this approach, we study the two most popular click models of OLTR, position-based model [4] and cascade model [5], and propose attack strategies against UCB-type OLTR algorithms. Their cumulative costs depend on κ K and p ∗ , which are instance-dependent parameters of the position-based and cascade models, respectively. Lastly, we introduce the threat model for OLTR with a general click model and design an attack algorithm that can misguide arbitrary OLTR algorithms using click feedback. Our technical contributions are summarized as follows.

We propose the new idea of conservative estimation of the target arm for attacking UCB in stochastic bandits with binary feedback, which resolves the issues caused by the bounded and discrete requirements of the post-attack feedback. This approach is also the backbone of other attack strategies in more complicated OLTR settings.

The PBM-UCB algorithm uses position bias-corrected counters rather than simple click counters to compute UCB indices; we provide the theoretical analysis of our attack algorithm against PBM-UCB by carefully treating these unusual counters.

The partial feedback of OLTR with the cascade click model brings a new challenge to the attack design. We provide a new regret-based analysis of our attack algorithm against CascadeUCB without suffering from the partial feedback issue.

We devise a general attack strategy based on a new probabilistic attack design . It can successfully attack arbitrary OLTR algorithms without knowing the details of the algorithm.

We also conduct experiments on both synthetic and real-world data to evaluate our proposed attack strategies. Experimental results show that they can effectively attack the corresponding OLTR algorithms with less cost compared to other baselines. Due to space constraints, proofs and empirical results are included in the appendix.

Related Work . Online learning to rank with different feedback models has attracted much attention in recent years. Though there are other types of feedback models such as topK feedback [13], click feedback has been widely used in literature. [4, 14, 15] consider the position-based model (PBM), where each position of the item list has an examination probability known or unknown by the learning agent. The cascade model in [5, 16, 17, 18, 19, 20] considers the setting where the user would check the recommended items sequentially and stop at the first clicked one; all items after the clicked item will not be examined. The dependent click model (DCM) is a generalization of the cascade model where the user may click on multiple items [21, 22]. There are also general click models [1, 2, 3]

that can cover some of the previous click models. In this paper, we mainly focus on attack design on PBM and cascade models; both of them adopt bounded and discrete click feedback thus require a new design other than previous works like [9] that can only handle unbounded and continuous feedback.

Adversarial attacks on different types of multi-armed bandit problems have been studied recently [9, 10, 23, 24, 25]. [9] proposes the first study of adversarial attacks on the classical stochastic bandit problem. It designs effective attack strategies against the glyph[epsilon1] -Greedy and UCB algorithms. [10] extends it to a more general setting, where the algorithm of the learning agent can be unknown. [23] studies adversarial attacks on linear contextual bandits where the adversarial modifications can be added to either rewards or contexts. To the best of our knowledge, we are the first to study adversarial attacks on OLTR where, in addition to the challenges raised by click feedback, the combinatorial action space and the censored feedback of OLTR make it non-trivial to design efficient attack strategies.

2 Preliminaries

In this section, we briefly discuss the three problem settings we consider in this paper.

Stochastic L -armed bandit . We consider an arm set [ L ] = { 1 , 2 , · · · , L } , with µ i as the expected reward of arm i . Without loss of generality, we assume µ 1 ≥ µ 2 ≥ · · · ≥ µ L . In round t , the player chooses an arm a t ∈ [ L ] to play and receives a reward r 0 t as feedback. In the click feedback setting, the realized reward r 0 t ∈ { 0 , 1 } of arm a t is sampled from a Bernoulli distribution with expectation µ a t . The player aims to find an optimal policy to maximize the long-term cumulative reward.

OLTR with position-based model [4] . This setting also considers the item (arm) set [ L ] where µ i represents the click probability of item i . However, in OLTR with the position-based model (PBM), in each round t , the player chooses an ordered list of K items, a t = ( a 1 ,t , · · · , a K,t ) , with known examination probability κ k for the k -th position in the list (assuming κ 1 ≥ · · · ≥ κ K ). The player then observes the click feedback of the chosen list from the user, denoted as r 0 t = ( r 0 1 ,t , · · · , r 0 K,t ) ∈ { 0 , 1 } K , where r 0 i,t is sampled from a Bernoulli distribution with expectation κ i µ a i,t . The reward obtained by the player is the sum of the clicks, i.e., ∑ K k =1 r 0 k,t . The goal of the player is to find an optimal policy that can maximize the long-term cumulative user clicks.

glyph[negationslash]

OLTR with cascade model [5] . Here, we consider the same item set [ L ] as that in the PBM model. For OLTR with cascade model, in each round t , the player chooses an ordered list of K items, a t = ( a 1 ,t , · · · , a K,t ) , for the user. The user then checks the list from a 1 ,t to a K,t , with probability µ a k,t to click the k -th item. She immediately stops at the first clicked item, and returns the click result back to the player. We denote the position of the first clicked item as C t ( C t = ∞ if no item was clicked). The click feedback of the player is r 0 t = ( r 0 1 ,t , · · · , r 0 K,t ) ∈ { 0 , 1 } K , where only r 0 C t ,t = 1 and r 0 k,t = 0 for all k = C t . The reward obtained by the player is again the sum of user clicks ∑ K k =1 r 0 k,t , but in the cascade model, it is at most 1 . The goal of the player is also to find an optimal policy that can maximize the long-term cumulative user clicks.

3 Attacks on Stochastic L-armed Bandits with Binary Feedback

As mentioned in the introduction, one main challenge of attacking OLTR algorithms comes from the binary click feedback: such binary feedback limits the possible actions of the attacker since they need to ensure the post-attack reward feedback is still valid (binary). This is a common issue for all OLTR with click feedback. Hence, in this section, we focus on adversarial attacks on the L -armed bandit problem with binary feedback. We propose an attack algorithm against the UCB algorithm, which is the backbone of the attack strategies for more complicated OLTR settings.

3.1 Threat Model

We first introduce the threat model for the L -armed bandit problem with binary feedback. In each round t , the player chooses an arm a t ∈ [ L ] to play. The environment generates the pre-attack reward feedback r 0 t ∈ { 0 , 1 } based on a Bernoulli distribution with mean µ a t . The attacker then observes a t , r 0 t , and decides the post-attack feedback r t ∈ { 0 , 1 } . The player only receives r t as the feedback and uses that to decide the next arm to pull, a t +1 , for round t +1 . Without loss of generality, we

glyph[negationslash]

glyph[negationslash]

assume arm L is a sub-optimal target arm. The attacker's goal is to misguide the player to pull the target arm L very often while using small attack costs. Let N i ( t ) denote the number of pulls of arm i up to round t . We say the attack is successful after T rounds if N L ( T ) = T -o ( T ) in expectation or with high probability, while the cumulative attack cost C ( T ) = ∑ T t =1 | r t -r 0 t | = o ( T ) .

3.2 Attack Algorithm against UCB

For a better illustration of the attack strategy, we first define the following auxiliary notations. Let τ i ( t ) := { s : s ≤ t, a s = i } be the set of rounds up to t when arm i is played. We denote the pre-attack average reward of arm i up to round t by ˆ µ 0 i ( t ) := N i ( t ) -1 ∑ s ∈ τ i ( t ) r 0 s . Lastly, let ˆ µ i ( t ) := N i ( t ) -1 ∑ s ∈ τ i ( t ) r s be the post-attack average reward of arm i up to round t .

As in [9], we consider attacks against the ( α, ψ ) -UCB algorithm from [12], where α = 4 . 5 and ψ : λ ↦→ λ 2 / 8 since Bernoulli random variables are 1 / 4 -sub-Gaussian. The original attack algorithm in [9] calculates an attack value α t for round t such that

where β ( N ) := √ 1 2 N log π 2 LN 2 3 δ ; ∆ 0 > 0 and δ > 0 are the parameters of the attack algorithm. The attacker then gives the post-attack reward r t = r 0 t -α t back to the player. The main idea of the attack algorithm in [9] is to compute the attack value so that the post-attack empirical estimates of the non-target arms are always less than those of the target arm.

However, this attack design only works when the reward space is unbounded and continuous, while in the threat model with binary feedback, the Bernoulli rewards are discrete and bounded . There are two issues raised by the discrete and bounded requirements. First, the calculated attack value α t is a real number, which may not make the post-attack reward feedback r t to be valid (binary). Second, in order to ensure that Eq. (1) is true, the calculated α t might be larger than 1, which is higher than the maximum attack value in our threat model. In other words, it is impossible to let Eq. (1) hold for all rounds, while such a guarantee was essential for the theoretical analysis in [9]. To overcome these issues, we propose a new attack algorithm against UCB on L -armed bandits with binary feedback. It is described in Algorithm 1. The algorithm maintains timestamp h a ( t ) for each arm a . In round t , if the arm pulled by the player, a t , is not the target arm L , it first checks the condition γ t ≤ r 0 t (Line 6), with γ t computed as

where [ z ] + = max(0 , z ) and µ L ( t ) := ˆ µ L ( t ) -2 β ( N L ( t )) . In fact, condition γ t ≤ r 0 t is equivalent to checking whether there exists a feasible α t to ensure Eq. (1) holds: if γ t ≤ r 0 t , with α t set to be glyph[ceilingleft] γ t glyph[ceilingright] (Line 7), Eq. (1) will hold in round t . Using a similar analysis as in [9], we can derive an upper bound of N a t ( t ) and prove the success of the attack up to round t . The algorithm also updates the timestamp h a t ( t ) = t for a t (line 7). If γ t > r 0 t , it indicates that there is no feasible α t that can

ensure Eq. (1). Instead, the algorithm sets α t = glyph[ceilingleft] ˜ γ t glyph[ceilingright] with ˜ γ t computed as

where h a t ( t -1) records the last round that Eq. (1) was satisfied. We can prove such an α t is always feasible ( α t ≤ r 0 t ) and it can ensure that

This new inequality always holds for all rounds, which helps guarantee the success of the attack.

Remark . Compared with Eq. (1), Eq. (4) uses a more conservative lower bound of µ L , µ L ( h a t ( t -1)) , instead of µ L ( t ) , on the right-hand side of the inequality. We refer to this as conservative estimation for the target arm L with respect to arm a t , where the estimated lower bound of L will only be updated when there exists feasible α t to ensure Eq. (1). Intuitively, it makes the attack algorithm more conservative and less likely to use a small attack value that may result in an invalid large attack value in later rounds. We use an inductive proof to show that there always exists α t such that Eq. (4) holds while keeping r t valid (binary). Therefore, conservative estimation addresses the problems posed by binary feedback. This forms the foundation for the attack algorithms tailored for OLTR with click feedback.

3.3 Analysis

We first show that our attack algorithm always returns valid binary feedback to the player.

Lemma 1. The post-attack feedback of Algorithm 1 is always valid, i.e., r t ∈ { 0 , 1 } for any t .

The proof of Lemma 1, which uses an inductive analysis on γ t , ˜ γ t , can be found in the appendix.

Define ∆ a := µ a -µ L . We give the following theorem to show the successful attack of Algorithm 1. Theorem 1. Suppose T ≥ L and δ ≤ 1 / 2 . With probability at least 1 -δ , Algorithm 1 misguides the UCB algorithm to choose the target arm L at least T -( L -1) ( 1 + 3 ∆ 2 0 log T ) rounds, using a cumulative attack cost at most

As T goes to infinity, we have

Compared with Theorem 2 in [9], the β term in our cost bound depends on log h a ( T ) instead of log T . Since β is a decreasing function and h a ( T ) ≤ T , our finite-time cost bound can be larger than that in [9]. However, our asymptotic analysis of the cost suggests that when T is large, such difference becomes negligible. Notice that the attack algorithm in [9] does not have any theoretical guarantee in the binary feedback setting, so this comparison is only meant to show the additional cost potentially caused by the conservative estimation.

4 Attacks on Online Learning to Rank

We now move to developing effective attack algorithms for more complicated OLTR settings. Since attacking OLTR also faces the challenges caused by binary click feedback, the attack algorithms in this section rely on our attack design for stochastic bandits with binary feedback in Section 3.

4.1 Position-Based Click Model

Threat Model . We introduce the threat model for online stochastic ranking with position-based click feedback. In each round t , the player chooses a list of K item, a t = ( a 1 ,t , · · · , a K,t ) to recommend. The environment generates the pre-attack click feedback r 0 t = ( r 0 1 ,t , · · · , r 0 K,t ) ∈ { 0 , 1 } K where r 0 i,t

Algorithm 2 Attack against the PBM-UCB algorithm

10: 11: 12: Algorithm 3 CAL_ALPHA 1: Input : item l , click feedback r 0 , item set a ∗ 2: for a ∈ a ∗ do 3: Calculate γ t ( l, a ) , ˜ γ t ( l, a ) according to Eqs. (6) and (7) 4: end for 5: γ max = max a ∈ a ∗ γ t ( l, a ) , ˜ γ max = max a ∈ a ∗ ˜ γ t ( l, a ) 6: if γ max ≤ r 0 then 7: α = glyph[ceilingleft] γ max glyph[ceilingright] , h l,a ( t ) = t for all a ∈ a ∗ 8: else 9: α = glyph[ceilingleft] ˜ γ max glyph[ceilingright] , h l,a ( t ) = h l,a ( t -1) for all a ∈ a ∗ 10: end if 11: Return α

is sampled from a Bernoulli distribution with mean κ i µ a i,t . The attacker then observes a t and r 0 t , and decides the post-attack click feedback r t ∈ { 0 , 1 } K . The player only receives r t as the feedback and uses it to decide the next list to recommend, a t +1 , for round t +1 . Without loss of generality, we assume item L is a sub-optimal target item. Similar to Section 3.1, we say the attack is successful after T rounds if the number of target item recommendations is N L ( T ) = T -o ( T ) in expectation or with high probability, while the cumulative attack cost C ( T ) = ∑ T t =1 || r t -r 0 t || 1 = o ( T ) .

Attack against PBM-UCB . We consider the PBM-UCB algorithm in [4] as the online ranking algorithm of the player, which computes the UCB index of each item a as

where ˜ N a ( t ) := ∑ t s =1 ∑ K i =1 κ i I { a i,s = a } is the position bias-corrected counter, ˆ µ a ( t ) is the empirical mean of item a , and glyph[epsilon1] is a parameter of the algorithm. The algorithm then chooses the corresponding first K items with the highest UCB indices as the recommendation list.

We propose our attack algorithm against PBM-UCB in Algorithm 2. It works by first randomly taking K -1 items out and making them a set with the target item L , denoted as a ∗ = { a ∗ 1 , · · · , a ∗ K -1 , L } . Then, based on the conservative estimation idea from Algorithm 1, it maintains a timestamp h ( l, a ) for each item l with respect to each a ∈ a ∗ . The intuition is that, to ensure a similar inequality as Eq. (4) for all rounds, we need to make conservative estimation on the lower bounds of µ a for all a ∈ a ∗ . This is handled by Algorithm 3, which maintains the timestamps h l,a ( t ) for the input item l and outputs the appropriate attack value α on l that can always ensure the required inequality. The value of parameters γ t ( l, a ) and ˜ γ t ( l, a ) in Algorithm 3 are computed as

glyph[negationslash]

Notice that Algorithm 2 could also work when there are more than one but less than K +1 target arms (the goal of the attacker becomes misguiding the player to recommend all target arms very often with sublinear cost). The only modification required is to put all of these target arms into a ∗ .

Analysis . The following theorem shows the attack against PBM-UCB is successful.

Theorem 2. Suppose T ≥ L and δ ≤ 1 / 2 . With probability at least 1 -δ , Algorithm 2 misguides the PBM-UCB algorithm to recommend the target item L at least T -( L -K ) ( 1+ glyph[epsilon1] 2 κ 2 K ∆ 2 0 log T ) rounds, using a cumulative attack cost at most

When T goes to infinity, we have

Proof sketch . Whenever a i,t / ∈ a ∗ is chosen by the player, there must exist a ∈ a ∗ such that ¯ µ a i,t ( t ) ≥ ¯ µ a ( t ) . The output α i,t of Algorithm 3 would ensure ˆ µ a i,t ( t ) ≤ ˆ µ a ( h a i,t ,a ( t -1)) -2 β ( N a ( h a i,t ,a ( t -1))) -∆ 0 , owing to the conservative estimations in γ t ( a i,t , a ) and ˜ γ t ( a i,t , a ) . Combining these two inequalities, we can get B a i,t ( t ) -B a ( t ) ≥ ∆ 0 . With a careful calculation on this inequality and the involved bias-corrected counters, we have N a i,t ( t ) ≤ ( 1+ glyph[epsilon1] 2 κ 2 K ∆ 2 0 log t ) . This result holds for any a i,t / ∈ a ∗ . Thus, we immediately get the bound of N L ( t ) . The remaining proof for the cost bound will be similar to that of Theorem 1.

Compared with Theorem 1, the asymptotic cost of Algorithm 2 has an additional dependency on 1 /κ 2 K , which suggests it may require more cost to achieve a successful attack in the PBM model, though the cost dependency on T is still logarithmic.

4.2 Cascade Click Model

Threat Model . We introduce the threat model for OLTR with cascade click feedback. In each round t , the player chooses a list of K items, a t = ( a 1 ,t , · · · , a K,t ) to recommend. The environment generates the pre-attack click feedback r 0 t = ( r 0 1 ,t , · · · , r 0 K,t ) ∈ { 0 , 1 } K , || r 0 t || ≤ 1 . Let C t denote the position of the clicked item, i.e., r 0 C t ,t = 1 ( C t = ∞ if no item was clicked). The attacker observes a t , r 0 t , C t , and decides the post-attack click feedback r t ∈ { 0 , 1 } K , || r t || 1 ≤ 1 . The player only receives r t as the feedback and uses it to decide the next list to recommend, a t +1 , for round t +1 . The goal of the attacker in this setting is the same as that in the PBM model.

Attack against CascadeUCB. We propose an attack algorithm against CascadeUCB in Algorithm 4. Similar to the attack against PBM-UCB, it first randomly generates a set of items a ∗ = { a ∗ 1 , · · · , a ∗ K -1 , L } . It also follows the idea of conservative estimation: when the clicked item a C t ,t does not belong to a ∗ , it calls Algorithm 3 to maintain h a C t ,t ,a for all a ∈ a ∗ and compute the attack value α C t ,t based on the conservative estimations. If the output α C t ,t = 1 , which means

Algorithm 5 Attack against arbitrary algorithm

the algorithm sets the clicked position to be zero, the player will keep checking the positions after C t . Since the pre-attack feedback of all items after position C t is zero, we need to find the first item a i,t ∈ a ∗ after position C t and set α i,t = -1 ( r i,t = 1 ). The empirical means of the items between position C t and i that are not in a ∗ decreases, and the empirical mean of a i,t ∈ a ∗ increases. Thus, it does not affect the success of the attack while still making the post-attack feedback r t valid.

Analysis . First, note that there is a mismatch between recommendation and observation in the cascade model: for all i such that C t < i ≤ K , a i,t is recommended but not observed, i.e., there is no new click sample for item a i,t for estimation. We can still follow a similar proof of Theorem 2 to get the upper bound of the number of observations (samples) for a i,t / ∈ a ∗ , but it is less than the number of recommendations and thus cannot ensure the success of the attack. To tackle this problem, we use a new regret-based analysis on the expected number of recommendations. We can also prove that the post-attack feedback of Algorithm 4 is always valid. Define p ∗ := ∏ K -1 i =1 µ i . We give the following theorem of the successful attack against CascadeUCB.

Theorem 3. Suppose T ≥ L and δ ≤ 1 / 2 . With probability at least 1 -δ , Algorithm 2 misguides the CascadeUCB algorithm to choose the target arm at least T -( L -K ) ( 12 p ∗ ∆ 2 0 log T ) rounds in expectation. Its cumulative attack cost is at most

As T goes to infinity, we have

Proof sketch . As mentioned above, we use a new regret-based analysis. The intuition is that the regret caused by any suboptimal item is the product of its reward gap with respect to the optimal list and its expected number of recommendations. If we know the regret upper bound and the reward gap lower bound, we can derive the upper bound of the expected number of recommendations. To do so, we first show that the post-attack problem can be viewed as a problem with a known reward gap lower bound. This can be obtained by ˆ µ a i,t ( t ) ≤ ˆ µ L ( h L i,t ,a ( t -1)) -2 β ( N a ( h a i,t ,L ( t -1))) -∆ 0 for any a i,t / ∈ a ∗ , which indicates the post-attack expected reward gap between a i,t and L is always larger than ∆ 0 . Then, the lower bound of the reward gap of a i,t with respect to the optimal list is p ∗ ∆ 0 . Based on the regret upper bound 12 log T/ ∆ 0 given in [5], we can get the upper bound of the expected number of recommendations 12 p ∗ ∆ 2 0 log T . For the remaining cost analysis, since the attack cost only depends on the observed items, we can still follow the proof of Theorem 2.

4.3 General Attacks on OLTR with General Click Model

We have provided attack strategies against UCB-based OLTR algorithms under two specific click models. A natural follow-up question is whether there exists an attack strategy that can attack any

OLTR algorithm under the general click model. To answer this question in what follows, we design an attack strategy that can misguide any OLTR algorithm without knowing the underlying algorithm. However, it may pay more cost (still sublinear) than the others since it cannot take advantage of the details of the algorithm to make fine-tuned adjustments.

Threat Model . We consider the threat model for OLTR with general click feedback. In each round t , the player chooses a list of K items, a t = ( a 1 ,t , · · · , a K,t ) to recommend. The environment generates the pre-attack click feedback r 0 t = ( r 0 1 ,t , · · · , r 0 K,t ) ∈ { 0 , 1 } K , r 0 t ∈ R c , where R c is the feasible feedback space of click model c . The attacker observes a t , r 0 t , and decides the post-attack click feedback r t ∈ { 0 , 1 } K , r t ∈ R c . The attack should be aware of R c ; otherwise, ensuring valid post-attack feedback is impossible. The player only receives r t as the feedback and uses it to decide the next list to recommend, a t +1 . The goal of the attacker is the same as that in the PBM model.

General Attack against Arbitrary OLTR Algorithm . We propose an attack algorithm against arbitrary OLTR algorithms in Algorithm 5. Similar to the attack against PBM-UCB, it first randomly generates a set of items a ∗ = { a ∗ 1 , · · · , a ∗ K -1 , L } . In each round, for each clicked item a i,t / ∈ a ∗ , the algorithm calculates an attack probability p i,t and uses that to decide whether its feedback needs to be changed to unclicked ( r i,t = 0 ). We prove that p i,t is actually an estimated upper bound of ∆ i /µ i , thus by such probabilistic feedback perturbations, the algorithm makes all items outside a ∗ be worse than the items inside a ∗ , which guarantees the success of the attack. We consider a general assumption for the OLTR algorithm to be attacked.

Assumption 1. The OLTR algorithm chooses suboptimal items no more than R ( T ) = o ( T ) times for T rounds,

Notice that both PBM-UCB and CascadeUCB satisfy the assumption with R ( T ) = O (log T ) . We give the following theorem to show the success of our general attack algorithm.

Theorem 4. Suppose T ≥ L and δ ≤ 1 / 2 . With probability at least 1 -δ , Algorithm 5 misguides arbitrary OLTR algorithm that satisfies Assumption 1 to choose the target item at least T -R ( T ) rounds in expectation. Its cumulative attack cost is at most

Compared with the results in the PBM and cascade models, the β () term in the cumulative cost of Algorithm 5 is β (1) , which can be much larger than the others. Also, the asymptotic costs of Algorithm 2 and Algorithm 4 is independent of β , while the asymptotic cost of Algorithm 5 depends on β , showing that Algorithm 5 is more costly than attack strategies specific to OLTR algorithms.

5 Experiments

We conduct experiments using both synthetic and real data (MovieLens 20M dataset [26]). Due to space limitations, we report only the results of OLTR using the position-based model. We use glyph[epsilon1] = 0 . 1 for the PBM-UCB algorithm. For the synthetic data, we take L = 16 , K = 8 , T = 100 , 000 ; { µ i } L i =1 are sampled from uniform distribution U (0 , 1) for Figure 1a, and from U (0 , x ) for Figure 1b. For the real data, we take L = 100 , K = 10 , T = 100 , 000 ; { µ i } L i =1 are extracted according to [2].

Using synthetic data, we first study how the algorithm's performance is influenced by the algorithmic and problem parameters. As shown in Figure 1a, the cost decreases with an increase in ∆ 0 , aligning with our observations in Theorem 2. Figure 1b shows that the cost increases as x increases, which suggests that our algorithm pays more costs when ∆ a is large. We then compare our algorithm with two baselines: Trivial all attacks all arms except the target arm as long as the attack is valid; Trivial set first randomly takes K -1 arms out to generate a set a ∗ and then attacks all arms outside the set as long as the attack is valid. Figure 2 shows that Trivial all algorithm cannot successfully attack PBM-UCB even with linear costs. Our algorithm and Trivial set have similar performance on the chosen ratio of the target arm as shown in Figures 2a and 2c, which is the chosen time of the target arm divided by the current round. However, our algorithm pays 50% and 40% less cost than Trivial set in Figures 2b and 2d, respectively, which validates the necessity of our attack design.

(a) Cost as

∆

0

varies

(b) Cost as

x

varies

le3

2.0

o=0.1

X=1

△o=0.2

1.00

X=0.8

1.5

o=0.5

0.75

X=0.5

8

1.0

8

0.50

0.5

0.25

0.0

0.00

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

t

1e5

t

1e5

(a) Chosen ratio (syn.)

(b) Cost (syn.)

(c) Chosen ratio (real)

(d) Cost (real)

tio

0.75

1.00

1.5

1e2

Our attack

Trivial

Trivialset

ratio

1.0

0.8

3

1e3

sen

0.50

1.0

0.6

Ourattack

Trivialall

8

2

S

0.25

Our attack

Trivialal

0.5

Chos

0.4

0.2

Trivialset

1

Trivialall

Ourattack

0.00

Trivialset

0.0

Trivialset

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

t

1e5

t

1e5

t

1e5

t

1e5

6 Concluding Remarks

This paper presents the first study on adversarial attacks against OLTR with click feedback. We propose attack strategies that can successfully mislead various OLTR algorithms across multiple click models. One limitation of our work is that it cannot be applied to feedback models without clicks, such as the topK feedback [13]. Although we provide theoretical results for all these strategies, as discussed in Section 3.3, the finite-time cost results might be further improved by a more fine-grained analysis on h a ( t ) . Additionally, there is no known lower bound for the cumulative attack cost, even for the stochastic bandit setting in the literature, making it unclear whether our attack strategies are (asymptotically) order-optimal. This study opens up several future directions. One is to design attack strategies for other OLTR algorithms and feedback models. Lastly, our study on the vulnerability of existing OLTR algorithms inspires the design of robust OLTR algorithms against adversarial attacks.

Acknowledgments and Disclosure of Funding

Shuai Li is supported by National Key Research and Development Program of China (2022ZD0114804) and National Natural Science Foundation of China (62376154, 62006151, 62076161). M. Hajiesmaili's work is supported by CAREER-2045641, CNS-2102963, CNS2106299, and CPS-2136199. Wierman is supported by NSF grants CNS-2146814, CPS-2136197, CNS-2106403, NGSDI-2105648.

References

Masrour Zoghi, Tomas Tunys, Mohammad Ghavamzadeh, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. Online learning to rank in stochastic click models. In International conference on machine learning , pages 4199-4208. PMLR, 2017.

Shuai Li, Tor Lattimore, and Csaba Szepesvári. Online learning to rank with features. In International Conference on Machine Learning , pages 3856-3865. PMLR, 2019.

Tor Lattimore, Branislav Kveton, Shuai Li, and Csaba Szepesvari. Toprank: A practical algorithm for online stochastic ranking. Advances in Neural Information Processing Systems , 31, 2018.

Paul Lagrée, Claire Vernade, and Olivier Cappe. Multiple-play bandits in the position-based model. Advances in Neural Information Processing Systems , 29, 2016.

Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learning to rank in the cascade model. In International conference on machine learning , pages 767-776. PMLR, 2015.

Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning , pages 129-136, 2007.

Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval , 3(3):225-331, 2009.

Fatih Cakir, Kun He, Xide Xia, Brian Kulis, and Stan Sclaroff. Deep metric learning to rank. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1861-1870, 2019.

Kwang-Sung Jun, Lihong Li, Yuzhe Ma, and Jerry Zhu. Adversarial attacks on stochastic bandits. Advances in Neural Information Processing Systems , 31, 2018.

Fang Liu and Ness Shroff. Data poisoning attacks on stochastic bandits. In International Conference on Machine Learning , pages 4042-4050. PMLR, 2019.

Yinglun Xu, Bhuvesh Kumar, and Jacob D Abernethy. Observation-free attacks on stochastic bandits. Advances in Neural Information Processing Systems , 34:22550-22561, 2021.

Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

Sougata Chaudhuri and Ambuj Tewari. Online learning to rank with top-k feedback. The Journal of Machine Learning Research , 18(1):3599-3648, 2017.

Junpei Komiyama, Junya Honda, and Akiko Takeda. Position-based multiple-play bandit problem with unknown position bias. Advances in Neural Information Processing Systems , 30, 2017.

Beyza Ermis, Patrick Ernst, Yannik Stein, and Giovanni Zappella. Learning to rank in the position based model with bandit feedback. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 2405-2412, 2020.

Branislav Kveton, Zheng Wen, Azin Ashkan, and Csaba Szepesvari. Combinatorial cascading bandits. Advances in Neural Information Processing Systems , 28, 2015.

Shi Zong, Hao Ni, Kenny Sung, Nan Rosemary Ke, Zheng Wen, and Branislav Kveton. Cascading bandits for large-scale recommendation problems. Thirty-Second Conference on Uncertainty in Artificial Intelligence , 2016.

Shuai Li, Baoxiang Wang, Shengyu Zhang, and Wei Chen. Contextual combinatorial cascading bandits. In International conference on machine learning , pages 1245-1253. PMLR, 2016.

Daniel Vial, Sujay Sanghavi, Sanjay Shakkottai, and R Srikant. Minimax regret for cascading bandits. Advances in Neural Information Processing Systems , 2022.

Xutong Liu, Jinhang Zuo, Siwei Wang, John CS Lui, Mohammad Hajiesmaili, Adam Wierman, and Wei Chen. Contextual combinatorial bandits with probabilistically triggered arms. In International Conference on Machine Learning , pages 22559-22593. PMLR, 2023.

Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. Dcm bandits: Learning to rank with multiple clicks. In International Conference on Machine Learning , pages 12151224. PMLR, 2016.

Weiwen Liu, Shuai Li, and Shengyu Zhang. Contextual dependent click bandit algorithm for web recommendation. In Computing and Combinatorics: 24th International Conference, COCOON 2018, Proceedings 24 , pages 39-50. Springer, 2018.

Evrard Garcelon, Baptiste Roziere, Laurent Meunier, Jean Tarbouriech, Olivier Teytaud, Alessandro Lazaric, and Matteo Pirotta. Adversarial attacks on linear contextual bandits. Advances in Neural Information Processing Systems , 33:14362-14373, 2020.

Yuzhe Ma and Zhijin Zhou. Adversarial attacks on adversarial bandits. International Conference on Learning Representations , 2023.

Huazheng Wang, Haifeng Xu, and Hongning Wang. When are linear stochastic bandits attackable? In International Conference on Machine Learning , pages 23254-23273. PMLR, 2022.

Movielens 20m dataset. https://grouplens.org/datasets/movielens/20m/ . [Online].

Appendix

A Proofs

A.1 Proof of Lemma 1

Proof. When γ t ≤ r 0 t , it is easy to check α t = glyph[ceilingleft] γ t glyph[ceilingright] ≤ r 0 t , thus r t ∈ { 0 , 1 } .

glyph[negationslash]

When γ t > r 0 t , if ˜ γ t ≤ r 0 t , we will have α t = glyph[ceilingleft] ˜ γ t glyph[ceilingright] ≤ r 0 t and r t ∈ { 0 , 1 } . Since ˜ γ t depends on a t , our goal is to show ˜ γ t ≤ r 0 t for any a t . Consider any arm a = K . We denote t a,j as the j -th time that the UCB algorithm played a . Since the UCB algorithm would play each arm one time in the beginning, we have

Next, we want to show that if ˜ γ t a,j ≤ r 0 t a,j , ˜ γ t a,j +1 ≤ r 0 t a,j +1 . Since ˜ γ t also depends on h a t ( t -1) , we consider two cases:

1) γ t a,j ≤ r 0 t a,j . In this case, we have h a ( t a,j +1 -1) = t a,j . We can bound ˜ γ t a,j +1 using γ t a,j :

where the second inequality is due to γ t a,j ≤ α t a,j .

2) γ t a,j > r 0 t a,j . In this case, we have h a ( t a,j +1 -1) = h a ( t a,j -1) . We can bound ˜ γ t a,j +1 using ˜ γ t a,j

where the second inequality is due to ˜ γ t a,j ≤ r 0 t a,j and ˜ γ t a,j ≤ α t a,j .

Since ˜ γ t a, 1 ≤ r 0 t a, 1 , by induction, we have ˜ γ t a,j ≤ r 0 t a,j for any a, j , which concludes the proof.

A.2 Proof of Theorem 1

Proof. We use the following two lemmas to prove Theorem 1. The proof is similar to those of Lemma 5 and Lemma 6 in [9], while as discussed in Section 3, the algorithm in [9] relies on Eq. (1) that may not hold due to the binary feedback, and our proof is mainly based on Eq. (4) that can be ensured by the conservative estimations. Define event E := {∀ i, ∀ t > L : | ˆ µ 0 i ( t ) -µ i | < β ( N i ( t )) } . With Hoeffding's inequality, it is easy to prove for δ ∈ (0 , 1) , P ( E ) > 1 -δ .

glyph[negationslash]

Lemma 2. Assume event E holds and δ ≤ 1 / 2 . For any a = L and any t ≥ L , we have

glyph[negationslash]

Proof. Fix some t ≥ L . We consider a t = a = L and denote t ′ = max { s < t : a s = a } as the previous round that a was pulled. Since arm a should be pulled at least once by the UCB algorithm, we have t ′ ≥ 1 , and the attack in round t ′ would ensure

Since arm a was pulled in round t , we know its UCB value must be greater than that of target arm L :

Considering that t ′ and t are two consecutive rounds when arm a was pulled, we have ˆ µ a ( t -1) = ˆ µ a ( t ′ ) and N a ( t -1) = N a ( t ′ ) . Rearranging Eq. (13), we get

where the second inequality comes from Equation (12). Since ∆ 0 > 0 , we have

Also, since 3 2 √ log( t ) N L ( t -1) > 0 , we have 3 2 √ log( t ) N a ( t ′ ) > ∆ 0 , which implies

Lemma 3. Assume event E holds and δ ≤ 1 / 2 .

1) For any t ≥ L , the cumulative attack cost to any fixed arm a = L can be bounded as:

glyph[negationslash]

2) When t goes to infinity, we have

Proof. Fix any arm a = L . By the definitions of γ t and ˜ γ t , it follows that:

glyph[negationslash]

where the last inequality is due to the decrease of β .

For the asymptotic result, combining Lemma 2 with Equation (17), we have

It is easy to check that lim t →∞ β ( t ) = 0 . Hence, to get the asymptotic cost bound in the lemma, we need to prove lim t →∞ β ( N L ( h a ( t ))) = 0 . We find lim t →∞ h a ( t ) = t is a sufficient condition for it; in other words, h a ( t ) should be always updated when t goes to infinity. To obtain this, we consider two cases of γ t .

1) If µ L ( t ) -∆ 0 ≥ 0 , we have

The first inequality is due to the attack at round t ′ . The last inequality is due to the confidence radius based on β ( N L ( h a ( t ′ ))) and β is decreasing.

2) If µ L ( t ) -∆ 0 < 0 , we have ˆ µ L ( t ) < 2 β ( N L ( t ))+∆ 0 and ˆ µ L ( h a ( t ′ )) ≤ ˆ µ L ( t )+2 β ( N L ( h a ( t ′ ))) .

Since in both cases, γ ≤ r , we have h ( t ) = t when t t 0 t a goes to infinity.

With Lemma 2 and Lemma 3, the proof is completed by summing the corresponding upper bounds over all non-target arms a < L ,

A.3 Proof of Theorem 2

Proof. As for Algorithm 1, we first need to prove the post-attack feedback of Algorithm 2 is always valid. It is equivalent to showing that the output α of Algorithm 3 is a valid attack value on the input pre-attack feedback r 0 , i.e., α ≤ r 0 . Similar to the proof of Lemma 1, we consider two cases: when γ max ≤ r 0 , α = glyph[ceilingleft] γ max glyph[ceilingright] ≤ r 0 ; when γ max < r 0 , we can use the same inductive proof of Lemma 1 to show that α = glyph[ceilingleft] ˜ γ max glyph[ceilingright] ≤ r 0 . Thus, the post-attack feedback r 0 -α is always valid.

Fix some t ≥ L such that a i,t / ∈ a ∗ . We denote t ′ = max { s < t : a i,t ∈ a s } as the previous round that a i,t was chosen. With the conservative estimations in γ t ( a i,t , a ) and ˜ γ t ( a i,t , a ) , the output α i,t of Algorithm 3 could ensure

for any a ∈ a ∗ . Since a i,t is chosen by the PBM-UCB algorithm, there must exist a ∈ a ∗ with its UCB value less than that of a i,t :

Since t ′ and t are two consecutive rounds when item a i,t was chosen, we have ˆ µ a i,t ( t -1) = ˆ µ a i,t ( t ′ ) , N a i,t ( t -1) = N a i,t ( t ′ ) . Rearranging Eq. (25), we have

where the last inequality is due to Eq. (24). Since B a i,t ( t ) ≥ ∆ 0 , with the definition of B a i,t ( t ) in Eq. (5), we have

for any a i,t / ∈ a ∗ . Thus, for the target item L ,

which guarantees the chosen time of the target item.

For the cumulative attack cost analysis, we consider any arm a i,t / ∈ a ∗ . Since Eq. (24) holds for any a ∈ a ∗ , we take a = L and get

where the second line is due to N a i,t ( t ) ≤ N L ( t ) , h a i,t ,L ( t ) ≤ N a i,t ( t ) , and β is decreasing. Based on this inequality, we can follow the same steps in the proof of Lemma 3 to derive the upper bound of the cumulative attack cost for Algorithm 2.

A.4 Proof of Theorem 3

Proof. Since Algorithm 4 also calls Algorithm 3, we can use the same proof in Appendix A.3 to show its post-attack feedback is always valid. Also, the output α i,t of Algorithm 3 would still ensure Eq. (24) for any a i,t / ∈ a ∗ , a ∈ a ∗ . Based on this, we can consider the post-attack problem as a cascading bandit problem with known expected reward gaps, where the expected reward of any item l / ∈ a ∗ is less than that of the target item L by at least ∆ 0 . Since L is the worst item in a ∗ , with the regret analysis of CascadeUCB in [5], we know the regret caused by any list containing l / ∈ a ∗ is bounded by

We also know the lower bound of the one-round instantaneous regret caused by any list containing l is p ∗ ∆ 0 , where p ∗ := ∏ K -1 i =1 µ i . We can then write the lower bound of R l ( T ) as

Combinining Eq. (31) and Eq. (32), we obtain the upper bound of the expected number of recommendations of item l :

Thus, for the target item L ,

For the remaining cost analysis, since the attack cost only depends on the observed items, we can still follow the proof of Theorem 2: we need to change the confidence radius B a ( t ) in Eq. (5) to the typical form 3 2 √ log t N a ( t -1) , then derive the corresponding cost upper bound.

A.5 Proof of Theorem 4

Proof. Since Algorithm 5 only attacks when r 0 i,t = 1 , it is easy to verify its post-attack feedback is valid. The key step for proving its successful attack is to show that p i,t is an upper bound of ∆ a i,t /µ a i,t : if it is true, by the probabilistic attack with probability p i,t , Algorithm 5 makes a postattack bandit problem where the expected reward of any item l / ∈ a ∗ is worse than that of target item L , which guarantees the number of target arm pulls is T -R ( T ) . Actually, we have

which increases as µ a i,t increases. Replacing µ a i,t with ˆ µ 0 a i,t ( t ) + β ( N a i,t ( t )) ,

where ˆ µ 0 a i,t ( t ) + β ( N a i,t ( t )) ≥ µ a i,t and ˆ µ 0 L ( t ) + β ( N L ( t )) ≥ µ L under event E . Since the last term is the definition of p i,t , we have proved it is a high-probability upper bound of ∆ a i,t /µ a i,t . For the cost analysis, we have

which concludes the proof.

B Additional Experiments

B.1 Synthetic Data

We first consider adversarial attacks on 2 -armed stochastic bandits with binary feedback. The rewards of arms 1 and 2 follow Bernoulli distributions with means µ 1 and µ 2 ( µ 1 > µ 2 ). We designate arm 2 as the target arm. We take T = 10 , 000 , δ = 0 . 1 , ∆ 0 = 0 . 1 . We set µ 1 = 0 . 85 and sample the value of µ 2 from a uniform distribution U (0 . 05 , 0 . 15) . We compare Algorithm 1 with a modified algorithm from [9], which attacks every non-target arm whenever the calculated α t > 0 and the pre-attack feedback is one. Note that this modified algorithm lacks a theoretical guarantee. We show the costs and target arm chosen times relative to a trivial baseline, which sets the post-attack feedback of all non-target arms to zero as long as the pre-attack feedback is one. Figure 3 demonstrates that our algorithm is more efficient than the modified one, incurring fewer costs for a greater number of target arm selections.

We also conduct experiments under the cascade model with synthetic data. We take L = 16 , K = 8 , T = 100 , 000 , and { µ i } L i =1 are sampled from a uniform distribution U (0 , 1) . We compare Algorithm 4 with the same baselines introduced in Section 5. Figure 4 shows that Trivial all algorithm suffers an extremely high cumulative cost, while it cannot misguide the agent to recommend the target arm very often. On the other hand, our algorithm and Trivial set algorithm can successfully attack CascadeUCB and perform similarly on the chosen ratio of the target arm. However, our algorithm pays about 30% less cost than Trivial set .

B.2 Real Data

As discussed in Section 5, we have shown results under the position-based model with real data. We now show the experimental results with real data (MovieLens 20M dataset) under the cascade model. We take L = 100 , K = 10 , T = 100 , 000 ; { µ i } L i =1 are extracted according to [2]. Similarly, we compare the chosen ratio and cumulative cost of our algorithm with Trivial all and Trivial set . Figure 5 shows that Trivial all cannot successfully attack CascadeUCB as the chosen ratio of the target arm is very low and it suffers a linear cost. Algorithm 4 and Trivial set have similar performance on the chosen ratio of the target arm. However, our algorithm dramatically decreases the cost by more than 50% , which indicates that it is more effective than Trivial set .

(a) Relative Cost

(b) Relative chosen times

le1

le1

0.25

1.5

Previous attack

Ourattack

0.00

S

D

1.0

-0.25

sen

0.5

S

-0.50

0

0

0.0

0.75

Ourattack

Previousattack

0.5

-1.08.25

0.50

0.75

1.00

1.25

1.50

0.25

0.50

0.75

1.00

1.25

1.50

μ2

1e1

μ2

1e1

(a) Cost

(b) Chosen times

1e2

1.0

1.0

0.8

atio

0.6

sen

rat

0.8

Trivialall

Our attack

0.4

S

0

Trivialset

Ourattack

0.6

0.2

Trivialall

Trivialset

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

t

1e5

t

1e5

(a) Cost

(b) Chosen times

1e3

1.00

1.25

1.00

0.75

ours

0.75

e

S

=

0.50

trivialset

trivialall

0.50

ours

0

0.25

0.25

trivialall

trivialset

0.00

0.00

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

t

1e5

t

1e5

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presented a study of reward poisoning attacks on UCB bandit algorithms with binary feedback and applies the designed attack to the online-learning-to-rank (OLTR) problem. The binary reward structure poses unique challenges to reward poisoning attacks, since the attacker needs to ensure the perturbed reward is still binary. The authors proposed an attack algorithm that achieves that. Based on that, the paper studied reward poisoning attacks in OLTR. The attacker is limited to change the click feedback, which is required to be binary before and after attack. Two threat models are considered - in the position-based click feedback scenario, the authors proposed attack strategy targeting the PBM-UCB algorithm; and in the cascade-click model, attack strategy against CascadeUCB is proposed. Both achieve T-O(log T) target item selections with only O(log T) attack cost. The paper also designed a general attack strategy for arbitrary click models that achieve O(log T) sub-optimal item selections. Finally, the paper performed empirical evaluations of the attacks.\n\nSTRENGTHS:\n(1). The paper first studied data poisoning attacks on ranking algorithms, which is timely topic following prior works that study reward poisoning attacks on multi-armed bandit. Although the idea behind the design of the attack algorithm is similar in nature, poisoning ranking algorithms poses unique challenges compared to bandits - the attack needs to ensure the poisoned reward is still binary. \n\n(2). The theoretical analysis of this paper is solid. First, an efficient attack targeting the UCB bandit algorithm is proposed, which guarantees that the poisoned reward is still binary. Then two theoretical results are presented to show that UCB-style ranking algorithms are susceptible to poisoning attacks in both position-click and cascade-click scenarios. Finally, the paper also provided a general attack for general click model ranking algorithms.\n\n(3). The paper performed extensive empirical evaluations of the proposed attack strategies. The results are convincing.\n\nWEAKNESSES:\n(1). The assumption made in section 4.3 is hidden in Theorem 4 - the ranking algorithm needs to guarantee O(log T) sub-optimal item selections. This assumption is critical and should not be hidden inside the theorem. Instead, the authors should consider carefully defining the assumption.\n\n(2). The Theorem 4 is not intriguing enough. In particular, it needs to assume that the ranking algorithm guarantees O(log T) sub-optimal item selections. Is O(log T) critical? Can it be O(sqrt(T))? The problem is that O(log T) seems too specific. Instead, it makes more sense to assume the ranking algorithm is no-regret, i.e., it selects suboptimal items in o(T) rounds. In that case, does Theorem 4 still hold?\n\nQUESTIONS:\n(1). Can the author make the assumption in Theorem 4 more prominent?\n\n(2). Is O(log T) sub-optimal item selections a critical assumption in Theorem 4? Can the authors extend the result to arbitrary no-regret ranking algorithms that guarantee o(T) suboptimal item selections?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces adversarial attack on online learning to rank. The attacker aims to fool the online learning to rank algorithm to choose the target item. The paper studied three victim models: stochastic L-armed bandit, OLTR with position-based model and OLTR with cascade model. Authors proposed three attack algorithm respectively for three victim models and a general attack strategy to attack arbitrary OLTR algorithms without knowing the details of the algorithm. Their theoretical analysis and experimental results show that the proposed attack strategies can manipulates the learning agent into choosing the target attack item in linear time but incurring a sublinear cost.\n\nSTRENGTHS:\nThe paper studied adversarial attacks on online learning to rank with click feedback, which as far as I know is the first work in this field. The paper provides some theoretical guarantees on their proposed algorithms. The results show that the attack strategies can successfully attack OLTR algorithms with sublinear cost. The paper is well-written and well-organized. The attack strategy on the UCB algorithm clearly shows that the differences between this work and previous work about reward attack on bandits.\n\nWEAKNESSES:\nOverall, the paper is well-written and has good technical content, but I have the following concerns:\n\n1. An intuitive attack method is that the attacker first chooses an action (list) $a$ including the target items and sets the reward out of this list to 0. The authors name this as $Trivial_{k-1}$. The authors empirically compared the proposed methods with $Trivial_{k-1}$, which is good. However,  $Trivial_{k-1}$ seems to keep the same cost scaling on time as the proposed attack strategies, which weakens the contribution of this work.\n\n2. There is no lower bound about the attack cost and any information. Compared with $Trivial_{k-1}$, The proposed attack algorithms improve the attack cost by a constant ratio. How far is the proposed attack strategies to the optimal?\n\nQUESTIONS:\nSee in weakness 2.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies adversarial attack to online learning to rank, where a successful attack to an OLTR algorithm is such that a suboptimal item is recommended for linear in T times while with log T cost. This paper approaches this attack problem by studying attack to discrete MAB as warmup and then moves on to attack UCB based OLTR algorithms under position and cascade models. Lastly, it ends up with an attack strategy for general algorithms and click models, both theoretical and simulation results are provided to justify the attacks being successful.\n\nSTRENGTHS:\n1. Adversarial attack to online learning to rank makes up the foundation for studying robust/trustworthy recommendation systems. I agree some research into this topic is a must-do.\n2. This paper is well written and easy to read. It appears to have a thoughtful path to break down the OLTR attack problem in a few pieces and tackle it step by step, which is good. However, I doubt, which is also my major concern for this paper, that are some of the steps necessary to derive a successful attack? See details in weakness I list below.\n\nWEAKNESSES:\n**Possibly overcomplicates the problem**\n\nThere’s another work [1]https://arxiv.org/pdf/2305.19218.pdf on arxiv that seems to study the same problem as this paper. This paper retains novelty as these two can be considered as concurrent works according to the arxiv one’s submission date. But interestingly, [1] seems to have conflicted results from this paper:\n\nThe major tension lies in this paper proposes to alter clicks of unwanted arms with a sophisticated probability (e.g. in Alg.5), while [1] simply set 1 to 0 when an unwanted arm has a click. Moreover, this paper tests in simulation a baseline attack Trivial_1, which is close to the attack proposed by [1] (except for  [1] has a cutoff time for attack), and this paper claims Trivial_1 fails to attack according to the plot. And not surprisingly, theorem and simulation results in [1] supports their attack. \n\nSo this finding casts some doubt on this paper whether it overcomplicates the problem, as the attack proposed in [1] is way simpler with no need to compute a sophisticated probability to attack. Here are some clues in this paper that I think questionable:\n\n1. In Section 3.2, it claims that the challenge in applying previous MAB attack to discret MAB is $\\alpha_t$ in (1) can be invalid (>=1) when the altered reward should keep binary, which leads to all the effort in finding a new  $\\alpha_t$ with care. But simply setting all returns for unwanted arms to be 0 may also get the job done.\n2. Simulation section shows that Trivial_1 fails while Trivial_{k-1} succeeds,I don’t think Trivial_{k-1} has essential difference than Trivial_1 as in the special case k=2 they are identical. So maybe there’s something going wrong with the plots or interpretation.\n\nQUESTIONS:\nMajor question please refer to weakness 1.\n\nLIMITATIONS:\nIdentified Limitations are listed in Weakness and Question.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies attack strategies against UCB-based OLTR algorithms for position-based click model and cascade click model. Compared with prior works in adversarial attacks, OLTR problem leads to the additional challenge that the rewards are discrete and bounded, which requires different attack method design, i.e., to ensure the post-attack reward is valid. The authors further propose attack strategy that works for OLTR algorithms with general click models, at the expense of additional attack cost compared with the strategies designed for specific click models.\n\nSTRENGTHS:\nThis paper is well-written and seems to be technically sound.\n\nTo the best of my knowledge, this is the first work that provides rigorous solutions and analysis for attacking OLTR algorithms.\n\nThe designed attack strategy for discrete and bounded reward might be of independent interest, e.g., it may be applied to other problem settings with discrete reward.\n\nWEAKNESSES:\nI think a more intuitive (and self-contained) explanation, on why the use of conservative estimation can address the challenge caused by bounded reward, would be helpful. The current writing of Section 3.2 requires some knowledge of of [9] to fully understand the intuition and appreciate its novelty.\n\nQUESTIONS:\nCan the authors provide more explanation on why constructing the set $a^{\\star}$ by randomly selecting $K-1$ items, in addition to the target item $L$? Since the goal is to make target item $L$ appear in the recommended set $a_{t}$, why do we want to spend cost on making the learner recommend the $K-1$ randomly sampled items as well? Is it possible to simply making sure L replaces the last item in $a_{t}$?\n\nThe reason the current design (for single target item) seems to be sub-optimal is that, this algorithm can be directly applied to the case where there are more than 1 but less than K+1 target items, without incurring additional cost, i.e., the algorithm will need to spend cost in making the learner pull $K-1$ randomly selected items anyway.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}