Title: Online Corrupted User Detection and Regret Minimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: online learning

online corrupted user detection

clustering of bandits
Abstract: In real-world online web systems, multiple users usually arrive sequentially into the system. For applications like click fraud and fake reviews, some users can maliciously perform corrupted (disrupted) behaviors to trick the system. Therefore, it is crucial to design efficient online learning algorithms to robustly learn from potentially corrupted user behaviors and accurately identify the corrupted users in an online manner.  Existing works propose bandit algorithms robust to adversarial corruption. However, these algorithms are designed for a single user, and cannot leverage the implicit social relations among multiple users for more efficient learning. Moreover, none of them consider how to detect corrupted users online in the multiple-user scenario. In this paper, we present an important online learning problem named LOCUD to learn and utilize unknown user relations from disrupted behaviors to speed up learning, and identify the corrupted users in an online setting. To robustly learn and utilize the unknown relations among potentially corrupted users, we propose a novel bandit algorithm RCLUB-WCU. To detect the corrupted users, we devise a novel online detection algorithm OCCUD based on RCLUB-WCU's inferred user relations. We prove a regret upper bound for RCLUB-WCU, which asymptotically matches the lower bound with respect to $T$ up to logarithmic factors, and matches the state-of-the-art results in degenerate cases. We also give a theoretical guarantee for the detection accuracy of OCCUD. With extensive experiments, our methods achieve superior performance over previous bandit algorithms and high corrupted user detection accuracy.

Online Corrupted User Detection and Regret Minimization

Abstract

In real-world online web systems, multiple users usually arrive sequentially into the system. For applications like click fraud and fake reviews, some users can maliciously perform corrupted (disrupted) behaviors to trick the system. Therefore, it is crucial to design efficient online learning algorithms to robustly learn from potentially corrupted user behaviors and accurately identify the corrupted users in an online manner. Existing works propose bandit algorithms robust to adversarial corruption. However, these algorithms are designed for a single user, and cannot leverage the implicit social relations among multiple users for more efficient learning. Moreover, none of them consider how to detect corrupted users online in the multiple-user scenario. In this paper, we present an important online learning problem named LOCUD to learn and utilize unknown user relations from disrupted behaviors to speed up learning, and identify the corrupted users in an online setting. To robustly learn and utilize the unknown relations among potentially corrupted users, we propose a novel bandit algorithm RCLUB-WCU. To detect the corrupted users, we devise a novel online detection algorithm OCCUD based on RCLUB-WCU's inferred user relations. We prove a regret upper bound for RCLUB-WCU, which asymptotically matches the lower bound with respect to T up to logarithmic factors, and matches the state-of-the-art results in degenerate cases. We also give a theoretical guarantee for the detection accuracy of OCCUD. With extensive experiments, our methods achieve superior performance over previous bandit algorithms and high corrupted user detection accuracy.

1 Introduction

In real-world online recommender systems, data from many users arrive in a streaming fashion [4, 15, 2, 7, 35, 27, 26]. There may exist some corrupted (malicious) users, whose behaviors ( e.g. , click, rating) can be adversarially corrupted (disrupted) over time to fool the system [29, 30, 12, 10, 9]. These corrupted behaviors could disrupt the user preference estimations of the algorithm. As a result, the system would easily be misled and make sub-optimal recommendations [14, 23, 7, 41],

which would hurt the user experience. Therefore, it is essential to design efficient online learning algorithms to robustly learn from potentially disrupted behaviors and detect corrupted users in an online manner.

There exist some works on bandits with adversarial corruption [29, 9, 22, 5, 12, 16]. However, they have the following limitations. First, existing algorithms are initially designed for robust online preference learning of a single user. In real-world scenarios with multiple users, they cannot robustly infer and utilize the implicit user relations for more efficient learning. Second, none of them consider how to identify corrupted users online in the multiple-user scenario. Though there also exist some works on corrupted user detection [34, 6, 39, 28, 13], they all focus on detection with known user information in an offline setting, thus can not be applied to do online detection from bandit feedback.

To address these limitations, we propose a novel bandit problem ' Learning and Online Corrupted Users Detection from bandit feedback ' (LOCUD). To model and utilize the relations among users, we assume there is an unknown clustering structure over users, where users with similar preferences lie in the same cluster [8, 19, 21]. The agent can infer the clustering structure to leverage the information of similar users for better recommendations. Among these users, there exists a small fraction of corrupted users. They can occasionally perform corrupted behaviors to fool the agent [12, 29, 30, 9] while mimicking the behaviors of normal users most of the time to make themselves hard to discover. The agent not only needs to learn the unknown user preferences and relations robustly from potentially disrupted feedback, balance the exploration-exploitation trade-off to maximize the cumulative reward, but also needs to detect the corrupted users online from bandit feedback.

The LOCUD problem is very challenging. First, the corrupted behaviors would cause inaccurate user preference estimations, which could lead to erroneous user relation inference and sub-optimal recommendations. Second, it is nontrivial to detect corrupted users online since their behaviors are dynamic over time (sometimes regular while sometimes corrupted), whereas, in the offline setting, corrupted users' information can be fully represented by static embeddings and the existing approaches [18, 32] can typically do binary classifications offline, which are not adaptive over time.

We propose a novel learning framework composed of two algorithms to address these challenges.

RCLUB-WCU. To robustly estimate user preferences, learn the unknown relations from potentially corrupted behaviors, and perform high-quality recommendations, we propose a novel bandit algorithm ' Robust CLUstering of Bandits With Corrupted Users ' (RCLUB-WCU), which maintains a dynamic graph over users to represent the learned clustering structure, where users linked by edges are inferred to be in the same cluster. RCLUB-WCU adaptively deletes edges and recommends arms based on aggregated interactive information in clusters. We do the following to ensure robust clustering structure learning. (i) To relieve the estimation inaccuracy caused by disrupted behaviors, we use weighted ridge regressions for robust user preference estimations. Specifically, we use the inverse of the confidence radius to weigh each sample. If the confidence radius associated with user i t and arm a t is large at t , the learner is quite uncertain about the estimation of i t 's preference on a t , indicating the sample at t is likely to be corrupted. Therefore, we use the inverse of the confidence radius to assign minor importance to the possibly disrupted samples when doing estimations. (ii) We design a robust edge deletion rule to divide the clusters by considering the potential effect of corruptions, which, together with (i), can ensure that after some interactions, users in the same connected component of the graph are in the same underlying cluster with high probability.

OCCUD. To detect corrupted users online, based on the learned clustering structure of RCLUBWCU, we devise a novel algorithm named ' Online Cluster-based Corrupted User Detection ' (OCCUD). At each round, we compare each user's non-robustly estimated preference vector (by ridge regression) and the robust estimation (by weighted regression) of the user's inferred cluster. If the gap exceeds a carefully-designed threshold, we detect this user as corrupted. The intuitions are as follows. With misleading behaviors, the non-robust preference estimations of corrupted users would be far from ground truths. On the other hand, with the accurate clustering of RCLUB-WCU, the robust estimations of users' inferred clusters should be close to ground truths. Therefore, for corrupted users, their non-robust estimates should be far from the robust estimates of their inferred clusters.

We summarize our contributions as follows.

· We present a novel online learning problem LOCUD, where the agent needs to (i) robustly learn and leverage the unknown user relations to improve online recommendation qualities under the disruption of corrupted user behaviors; (ii) detect the corrupted users online from bandit feedback.

We propose a novel online learning framework composed of two algorithms, RCLUB-WCU and OCCUD, to tackle the challenging LOCUD problem. RCLUB-WCU robustly learns and utilizes the unknown social relations among potentially corrupted users to efficiently minimize regret. Based on RCLUB-WCU's inferred user relations, OCCUD accurately detects corrupted users online.

We prove a regret upper bound for RCLUB-WCU, which matches the lower bound asymptotically in T up to logarithmic factors and matches the state-of-the-art results in several degenerate cases. We also give a theoretical performance guarantee for the online detection algorithm OCCUD.

Experiments on both synthetic and real-world data clearly show the advantages of our methods.

2 Related Work

Our work is related to bandits with adversarial corruption and bandits leveraging user relations.

The work [29] first studies stochastic bandits with adversarial corruption, where the rewards are corrupted with the sum of corruption magnitudes in all rounds constrained by the corruption level C . They propose a robust elimination-based algorithm. The paper [9] proposes an improved algorithm with a tighter regret bound. The paper [22] first studies stochastic linear bandits with adversarial corruptions. To tackle the contextual linear bandit setting where the arm set changes over time, the work [5] proposes a variant of the OFUL [1] that achieves a sub-linear regret. A recent work [12] proposes the CW-OFUL algorithm that achieves a nearly optimal regret bound. All these works focus on designing robust bandit algorithms for a single user; none consider how to robustly learn and leverage the implicit relations among potentially corrupted users for more efficient learning. Moreover, none of them consider how to online detect corrupted users in the multiple-user case.

Some works study how to leverage user relations to accelerate the bandit learning process in the multiple-user case. The work [38] utilizes a known user adjacency graph to share context and payoffs among neighbors. To adaptively learn and utilize unknown user relations, the paper [8] proposes the clustering of bandits (CB) problem where there is an unknown user clustering structure to be learned by the agent. The work [20] uses collaborative effects on items to guide the clustering of users. The paper [19] studies the CB problem in the cascading bandit setting. The work [21] considers the setting where users in the same cluster share both the same preference and the same arrival rate. The paper [25] studies the federated CB problem, considering privacy and communication issues. All these works only consider utilizing the relations among normal users; none of them consider how to robustly learn the user relations from potentially disrupted behaviors, thus would easily be misled by corrupted users. Also, none of them consider how to detect corrupted users from bandit feedback.

To the best of our knowledge, this is the first work to study the problem to (i) learn the unknown user relations and preferences from potentially corrupted feedback, and leverage the learned relations to speed up learning; (ii) adaptively detect the corrupted users online from bandit feedback.

3 Problem Setup

This section formulates the problem of ' Learning and Online Corrupted Users Detection from bandit feedback ' (LOCUD) (illustrated in Fig.1). We denote ∥ x ∥ M = √ x ⊤ Mx , [ m ] = { 1 , . . . , m } , number of elements in set A as |A| .

̸

In LOCUD, there are u users, which we denote by set U = { 1 , 2 , . . . , u } . Some of them are corrupted users, denoted by set ˜ U ⊆ U . These corrupted users, on the one hand, try to mimic normal users to make themselves hard to detect; on the other hand, they can occasionally perform corrupted behaviors to fool the agent into making sub-optimal decisions. Each user i ∈ U , no matter a normal one or corrupted one, is associated with a (possibly mimicked for corrupted users) preference feature vector θ i ∈ R d that is unknown and ∥ θ i ∥ 2 ≤ 1 . There is an underlying clustering structure among all the users representing the similarity of their preferences, but it is unknown to the agent and needs to be learned via interactions. Specifically, the set of users U can be partitioned into m ( m ≪ u ) clusters, V 1 , V 2 , . . . V m , where ∪ j ∈ [ m ] V j = U , and V j ∩ V j ′ = ∅ , for j = j ′ . Users in the same cluster have the same preference feature vector, while users in different clusters have different preference vectors. We use θ j to denote the common preference vector shared by users in the j -th cluster V j , and use j ( i ) to denote the index of cluster user i belongs to ( i.e. , i ∈ V j ( i ) ). For

̸

any two users k, i ∈ U , if k ∈ V j ( i ) , then θ k = θ j ( i ) = θ i ; otherwise θ k = θ i . We assume the arm set A ⊆ R d is finite. Each arm a ∈ A is associated with a feature vector x a ∈ R d with ∥ x a ∥ 2 ≤ 1 .

The learning process of the agent is as follows. At each round t ∈ [ T ] , a user i t ∈ U comes to be served, and the learning agent receives a set of arms A t ⊆ A to choose from. The agent infers the cluster V t that user i t belongs to based on the interaction history, and recommends an arm a t ∈ A t according to the aggregated information gathered in the cluster V t . After receiving the recommended arm a t , a normal user i t will give a random reward with expectation x ⊤ a t θ i t to the agent.

To model the behaviors of corrupted users, following [29, 9, 5, 12], we assume that they can occasionally corrupt the rewards to mislead the agent into recommending sub-optimal arms. Specifically, at each round t , if the current served user is a corrupted user (i.e., i t ∈ ˜ U ), the user can corrupt the reward by c t . In summary, we model the reward received by the agent at round t as

1

8

7

6

5

4

3

2

1

8

7

6

5

4

3

2

GLYPH(cmap:d835)

GLYPH(cmap:d835)

1

8

7

6

5

4

3

2

GLYPH(cmap:d835)

GLYPH(cmap:d835)

1

8

7

6

5

4

3

2

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

where c t = 0 if i t is a normal user, ( i.e. , i t / ∈ ˜ U ), and η t is 1-sub-Gaussian random noise.

As the number of corrupted users is usually small, and they only corrupt the rewards occasionally with small magnitudes to make themselves hard to detect, we assume the sum of corruption magnitudes in all rounds is upper bounded by the corruption level C , i.e. , ∑ T t =1 | c t | ≤ C [29, 9, 5, 12].

We assume the clusters, users, and items satisfy the following assumptions. Note that all these assumptions basically follow the settings from classical works on clustering of bandits [8, 19, 25, 36].

Assumption 1 (Gap between different clusters) . The gap between any two preference vectors for different clusters is at least an unknown positive constant γ

̸

Assumption 2 (Uniform arrival of users) . At each round t , a user i t comes uniformly at random from U with probability 1 /u , independent of the past rounds.

Assumption 3 (Item regularity) . At each round t , the feature vector x a of each arm a ∈ A t is drawn independently from a fixed unknown distribution ρ over { x ∈ R d : ∥ x ∥ 2 ≤ 1 } , where E x ∼ ρ [ xx ⊤ ] 's minimal eigenvalue λ x > 0 . At ∀ t , for any fixed unit vector z ∈ R d , ( θ ⊤ z ) 2 has sub-Gaussian tail with variance no greater than σ 2 .

Let a ∗ t ∈ arg max a ∈A t x ⊤ a θ i t denote an optimal arm with the highest expected reward at round t . One objective of the learning agent is to minimize the expected cumulative regret

Another objective is to detect corrupted users online accurately. Specifically, at round t , the agent will give a set of users ˜ U t as the detected corrupted users, and we want ˜ U t to be as close to the ground-truth set of corrupted users ˜ U as possible.

4 Algorithms

This section introduces our algorithms RCLUB-WCU (Algo.1) and OCCUD (Algo.2). RCLUBWCU robustly learns the unknown user clustering structure and preferences from corrupted feed-

back, and leverages the cluster-based information to accelerate learning. Based on the clustering structure learned by RCLUB-WCU, OCCUD can accurately detect corrupted users online.

Algorithm 1 RCLUB-WCU

1: Input: Regularization parameter λ , confidence radius parameter β , threshold parameter α , edge deletion parameter α 1 , f ( T ) = √ (1 + ln(1 + T )) / (1 + T ) .

2: Initialization: M i, 0 = 0 d × d , b i, 0 = 0 d × 1 , ˜ M i, 0 = 0 d × d , ˜ b i, 0 = 0 d × 1 , T i, 0 = 0 , ∀ i ∈ U ; A complete graph G 0 = ( U , E 0 ) over U .

3: for all t = 1 , 2 , . . . , T do

4: Receive the index of the current served user i t ∈ U , get the feasible arm set at this round A t .

5: Determine the connected components V t in the current maintained graph G t -1 = ( U , E t -1 ) such that i t ∈ V t .

6: Calculate the robustly estimated statistics for the cluster V t : M V t ,t -1 = λ I + ∑ i ∈ V t M i,t -1 , b V t ,t -1 = ∑ i ∈ V t b i,t -1 , ˆ θ V t ,t -1 = M -1 V t ,t -1 b V t ,t -1 ;

7: Select an arm a t with largest UCB index in Eq.(3) and receive the corresponding reward r t ;

8: Update the statistics for robust estimation of user i t : M i t ,t = M i t ,t -1 + w i t ,t -1 x a t x ⊤ a t , b i t ,t = b i t ,t -1 + w i t ,t -1 r t x a t , T i t ,t = T i t ,t -1 +1 , M ′ i t ,t = λ I + M i t ,t , ˆ θ i t ,t = M ′-1 i t ,t b i t ,t , w i t ,t = min { 1 , α/ ∥ x a t ∥ M ′-1 i t ,t } ;

̸

9: Keep robust estimation statistics of other users unchanged: M ℓ,t = M ℓ,t -1 , b ℓ,t = b ℓ,t -1 , T ℓ,t = T ℓ,t -1 , ˆ θ ℓ,t = ˆ θ ℓ,t -1 , for all ℓ ∈ U , ℓ = i t ;

10: Delete the edge ( i t , ℓ ) ∈ E t -1 , if

and get an updated graph G t = ( U , E t ) ;

11: Use the OCCUD Algorithm (Algo.2) to detect the corrupted users.

12: end for

4.1 RCLUB-WCU

The corrupted behaviors may cause inaccurate preference estimations, leading to erroneous relation inference and sub-optimal decisions. In this case, how to learn and utilize unknown user relations to accelerate learning becomes non-trivial. Motivated by this, we design RCLUB-WCU as follows.

Assign the inferred cluster V t for user i t . RCLUB-WCU maintains a dynamic undirected graph G t = ( U , E t ) over users, which is initialized to be a complete graph (Algo.1 Line 2). Users with similar learned preferences will be connected with edges in E t . The connected components in the graph represent the inferred clusters by the algorithm. At round t , user i t comes to be served with a feasible arm set A t for the agent to choose from (Line 4). In Line 5, RCLUB-WCU detects the connected component V t in the graph containing user i t to be the current inferred cluster for i t .

Robust preference estimation of cluster V t . After determining the cluster V t , RCLUB-WCU estimates the common preferences for users in V t using the historical feedback of all users in V t and recommends an arm accordingly. The corrupted behaviors could cause inaccurate preference estimates, which can easily mislead the agent. To address this, inspired by [40, 12], we use weighted ridge regression to make corruption-robust estimations. Specifically, RCLUB-WCU robustly estimates the common preference vector of cluster V t by solving the following weighted ridge regression

where λ > 0 is a regularization coefficient. Its closed-form solution is ˆ θ V t ,t -1 = M -1 V t ,t -1 b V t ,t -1 , where M V t ,t -1 = λ I + ∑ s ∈ [ t -1] i s ∈ Vt w i s ,s x a s x ⊤ a s , b V t ,t -1 = ∑ s ∈ [ t -1] i s ∈ Vt w i s ,s r a s x a s .

We set the weight of sample for user i s in V t at round s as w i s ,s = min { 1 , α/ ∥ x a s ∥ M ′-1 i s ,s } , where α is a coefficient to be determined later. The intuitions of designing these weights are as follows. The term ∥ x a s ∥ M ′-1 i s ,s is the confidence radius of arm a s for user i s at s , reflecting how confident the algorithm is about the estimation of i s 's preference on a s at s . If ∥ x a s ∥ M ′-1 i s ,s is large, it means the agent is uncertain of user i s 's preference on a s , indicating this sample is probably corrupted.

Therefore, we use the inverse of confidence radius to assign a small weight to this round's sample if it is potentially corrupted. In this way, uncertain information for users in cluster V t is assigned with less importance when estimating the V t 's preference vector, which could help relieve the estimation inaccuracy caused by corruption. For technical details, please refer to Section 5.1 and Appendix.

Recommend a t with estimated preference of cluster V t . Based on the corruption-robust preference estimation ˆ θ V t ,t -1 of cluster V t , in Line 7, the agent recommends an arm using the upper confidence bound (UCB) strategy to balance exploration and exploitation

where β = √ λ + √ 2 log( 1 δ ) + d log(1 + T λd )+ αC is the confidence radius parameter, ˆ R a,t denotes the estimated reward of arm a at t , C a,t denotes the confidence radius of arm a at t . The design of C a,t theoretically relies on Lemma 2 that will be given in Section 5.

Update the robust estimation of user i t . After receiving r t , the algorithm updates the estimation statistics of user i t , while keeping the statistics of others unchanged (Line 8 and Line 9). Specifically, RCLUB-WCU estimates the preference vector of user i t by solving a weighted ridge regression

with closed-form solution ˆ θ i t ,t = ( λ I + M i t ,t ) -1 b i t ,t , where M i t ,t = ∑ s ∈ [ t ] i s = i t w i s ,s x a s x ⊤ a s , b i t ,t = ∑ s ∈ [ t ] i s = i t w i s ,s r a s x a s , and we design the weights in the same way by the same reasoning. Update the dynamic graph. Finally, with the updated statistics of user i t , RCLUB-WCU checks

Algorithm 2 OCCUD (At round t , used in Line 11 in Algo.1)

1: Initialize ˜ U t = ∅ ; input probability parameter δ .

2: Update the statistics for non-robust estimation of user i t

˜ M i t ,t = ˜ M i t ,t -1 + x a t x ⊤ a t , ˜ b i t ,t = ˜ b i t ,t -1 + r t x a t , ˜ θ i t ,t = ( λ I + ˜ M i t ,t ) -1 ˜ b i t ,t ,

3: Keep non-robust estimation statistics of other users unchanged

̸

:

˜ M ℓ,t = ˜ M ℓ,t -1 , ˜ b ℓ,t = ˜ b ℓ,t -1 , ˜ θ ℓ,t = ˜ θ ℓ,t -1 , for all ℓ ∈ U , ℓ = i t .

4: for all connected component V j,t ∈ G t do

5: Calculate the robust estimation statistics for the cluster V j,t M V j,t ,t = λ I + ∑ ℓ ∈ V j,t M ℓ,t , T V j,t ,t = ∑ ℓ ∈ V j,t T ℓ,t , b V j,t ,t = ∑ ℓ ∈ V j,t b ℓ,t , ˆ θ V j,t ,t = M -1 V j,t ,t b V j,t ,t ;

6: for all user i ∈ V j,t do

7: Detect user i to be a corrupted user and add user i to the set ˜ U t if the following holds:

where λ min ( · ) denotes the minimum eigenvalue of the matrix argument.

8: end for

9: end for

whether the inferred i t 's preference similarities with other users are still true, and updates the graph accordingly. Precisely, if gap between the updated estimation ˆ θ i t ,t of i t and the estimation ˆ θ ℓ,t of user ℓ exceeds a threshold in Line 10, RCLUB-WCU will delete the edge ( i t , ℓ ) in G t -1 to split them apart. The threshold is carefully designed to handle the estimation uncertainty from both stochastic noises and potential corruptions. The updated graph G t = ( U , E t ) will be used in the next round.

4.2 OCCUD

Based on the inferred clustering structure of RCLUB-WCU, we devise a novel online detection algorithm OCCUD (Algo.2). The design ideas and process of OCCUD are as follows.

Besides the robust preference estimations (with weighted regression) of users and clusters kept by RCLUB-WCU, OCCUD also maintains the non-robust estimations for each user by online ridge regression without weights (Line 2 and Line 3). Specifically, at round t , OCCUD updates the nonrobust estimation of user i t by solving the following online ridge regression:

With the robust and non-robust preference estimations, OCCUD does the following to detect corrupted users based on the clustering structure inferred by RCLUB-WCU. First, OCCUD finds the connected components in the graph kept by RCLUB-WCU, which represent the inferred clusters. Then, for each inferred cluster V j,t ∈ G t : (1) OCCUD computes its robustly estimated preferences vector ˆ θ V i,t ,t (Line 5). (2) For each user i whose inferred cluster is V j,t ( i.e., i ∈ V j,t ), OCCUD computes the gap between user i 's non-robustly estimated preference vector ˜ θ i,t and the robust estimation ˆ θ V i,t ,t for user i 's inferred cluster V j,t . If the gap exceeds a carefully-designed threshold, OCCUD will detect user i as corrupted and add i to the detected corrupted user set ˜ U t (Line 7).

1

8

7

6

5

4

3

2

1

8

7

6

5

4

3

2

1

8

7

6

5

4

3

2

(a) RCLUB-WCU

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

3

7

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

1

5

6

8

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

4

2

(b) OCCUD

The intuitions of OCCUD are as follows. On the one hand, after some interactions, RCLUB-WCU will infer the user clustering structure accurately. Thus, at round t , the robust estimation ˆ θ V i,t ,t for user i 's inferred cluster should be pretty close to user i 's ground-truth preference vector θ i . On the other hand, since the feedback of normal users are always regular, at round t , if user i is a normal user, the non-robust estimation ˜ θ i,t should also be close to the ground-truth θ i . However, the nonrobust estimation of a corrupted user should be quite far from the ground truth due to corruptions. Based on this reasoning, OCCUD compares each user's non-robust estimation and the robust estimation of the user's inferred cluster to detect the corrupted users. For technical details, please refer to Section 5.2 and Appendix. Simple illustrations of our proposed algorithms can be found in Fig.2.

5 Theoretical Analysis

In this section, we theoretically analyze the performances of our proposed algorithms, RCLUBWCUand OCCUD. Due to the page limit, we put the proofs in the Appendix.

5.1 Regret Analysis of RCLUB-WCU

This section gives an upper bound of the expected regret (defined in Eq.(1)) for RCLUB-WCU.

The following lemma provides a sufficient time T 0 ( δ ) , after which RCLUB-WCU can cluster all the users correctly with high probability.

Lemma 1. With probability at least 1 -3 δ , RCLUB-WCU will cluster all the users correctly after

After T 0 ( δ ) , the following lemma gives a bound of the gap between ˆ θ V t ,t -1 and the ground-truth θ i t in direction of action vector x a for RCLUB-WCU, which supports the design in Eq.(3).

Lemma 2. With probability at least 1 -4 δ for some δ ∈ (0 , 1 4 ) , ∀ t ≥ T 0 ( δ ) , we have:

With Lemma 1 and 2, we prove the following theorem on the regret upper bound of RCLUB-WCU. Theorem 3 ( Regret Upper Bound of RCLUB-WCU ) . With the assumptions in Section 3, and picking α = √ d + √ λ C , the expected regret of the RCLUB-WCU algorithm for T rounds satisfies

Discussion and Comparison. The regret bound in Eq.(7) has three terms. The first term is the time needed to get enough information for accurate robust estimations such that RCLUB-WCU could cluster all users correctly afterward with high probability. This term is related to the corruption level C , which is inevitable since, if there are more corrupted user feedback, it will be harder for the algorithm to learn the clustering structure correctly. The last two terms correspond to the regret after T 0 with the correct clustering. Specifically, the second term is caused by stochastic noises when leveraging the aggregated information within clusters to make recommendations; the third term associated with the corruption level C is the regret caused by the disruption of corrupted behaviors.

When the corruption level C is unknown , we can use its estimated upper bound ˆ C ≜ √ T to replace C in the algorithm. In this way, if C ≤ ˆ C , the bound will be replacing C with ˆ C in Eq.(7); when C > √ T , R ( T ) = O ( T ) , which is already optimal for a large class of bandit algorithms [12].

The following theorem gives a regret lower bound of the LOCUD problem.

Theorem 4 (Regret lower bound for LOCUD) . There exists a problem instance for the LOCUD problem such that for any algorithm

Its proof and discussions can be found in Appendix D. The upper bound in Theorem 3 asymptotically matches this lower bound in T up to logarithmic factors, showing our regret bound is nearly optimal.

We then compare our regret upper bound with several degenerated cases. First, when C = 0 , i.e. , all users are normal, our setting degenerates to the classic CB problem [8]. In this case the bound in Theorem 3 becomes O (1 / ˜ λ 2 x · u log( T )) + O ( d √ mT log( T )) , perfectly matching the state-ofthe-art results in CB [8, 19, 21]. Second, when m = 1 and u = 1 , i.e. , there is only one user, our setting degenerates to linear bandits with adversarial corruptions [22, 12], and the bound in Theorem 3 becomes O ( d √ T log( T )) + O ( Cd log 1 . 5 ( T )) , it also perfectly matches the nearly optimal result in [12]. The above comparisons also show the tightness of the regret bound of RCLUB-WCU.

5.2 Theoretical Performance Guarantee for OCCUD

The following theorem gives a performance guarantee of the online detection algorithm OCCUD.

Theorem 5 ( Theoretical Guarantee for OCCUD ) . With assumptions in Section 3, at ∀ t ≥ T 0 ( δ ) , for any detected corrupted user i ∈ ˜ U t , with probability at least 1 -5 δ , i is indeed a corrupted user.

This theorem guarantees that after RCLUB-WCU learns the clustering structure accurately, with high probability, the corrupted users detected by OCCUD are indeed corrupted, showing the high detection accuracy of OCCUD. The proof of Theorem 5 can be found in Appendix D.

6 Experiments

This section shows experimental results on synthetic and real data to evaluate RCLUB-WCU's recommendation quality and OCCUD's detection accuracy. We compare RCLUB-WCU to LinUCB [1] with a single non-robust estimated vector for all users, LinUCB-Ind with separate non-robust estimated vectors for each user, CW-OFUL [12] with a single robust estimated vector for all users, CW-OFUL-Ind with separate robust estimated vectors for each user, CLUB[8], and SCLUB[21]. More description of these baselines are in Appendix F. To show that the design of OCCUD is nontrivial, we develop a straightforward detection algorithm GCUD, which leverages the same cluster structure as OCCUD but detects corrupted users by selecting users with highest ∥ ∥ ∥ ˆ θ i,t -ˆ θ V i,t ,t -1 ∥ ∥ ∥ 2

in each inferred cluster. GCUD selects users according to the underlying percentage of corrupted users, which is unrealistic in practice, but OCCUD still performs better in this unfair condition.

Remark. The offline detection methods [39, 6, 18, 32] need to know all the user information in advance to derive the user embedding for classification, so they cannot be directly applied in online detection with bandit feedback thus cannot be directly compared to OCCUD. However, we observe the AUC achieved by OCCUD on Amazon and Yelp (in Tab.1) is similar to recent offline methods [18, 32]. Additionally, OCCUD has rigorous theoretical performance guarantee (Section 5.2).

6.1 Experiments on Synthetic Dataset

We use u = 1 , 000 users and m = 10 clusters, where each cluster contains 100 users. We randomly select 100 users as the corrupted users. The preference and arm (item) vectors are drawn in d -1 ( d = 50 ) dimensions with each entry a standard Gaussian variable and then normalized, added one more dimension with constant 1, and divided by √ 2 [21]. We fix an arm set with |A| = 1000 items, at each round, 20 items are randomly selected to form a set A t to choose from. Following [40, 3], in the first k rounds, we always flip the reward of corrupted users by setting r t = -x T a t θ i t ,t + η t . And we leave the remaining T -k rounds intact. Here we set T = 1 , 000 , 000 and k = 20 , 000 .

0.0

0.2

0.4

0.6

0.8

1.0



1e6

0

2

4

6

8



1e4















(a) Synthetic

0.0

0.2

0.4

0.6

0.8

1.0



1e6

0

1

2

3

4

5



1e4















(b) Movielens

0.0

0.2

0.4

0.6

0.8

1.0



1e6

0

1

2

3

4

5

6

7



1e4















(c) Amazon

0.0

0.2

0.4

0.6

0.8

1.0



1e6

0.0

0.2

0.4

0.6

0.8

1.0

1.2



1e5















(d) Yelp

Fig.3(a) shows the recommendation results. RCLUB-WCU outperforms all baselines and achieves a sub-linear regret. LinUCB and CW-OFUL perform worst as they ignore the preference differences among users. CW-OFUL-Ind outperforms LinUCB-Ind because it considers the corruption, but worse than RCLUB-WCU since it does not consider leveraging user relations to speed up learning.

The detection results are shown in Tab.1. We test the AUC of OCCUD and GCUD in every 200 , 000 rounds. OCCUD's performance improves over time with more interactions, while GCUD's performance is much worse as it detects corrupted users only relying on the robust estimations. OCCUD finally achieves an AUC of 0.855, indicating it can identify most of the corrupted users.

6.2 Experiments on Real-world Datasets

We use three real-world data Movielens [11], Amazon[31], and Yelp [33]. The Movielens data does not have the corrupted users' labels, so following [24], we manually select the corrupted users. On Amazon data, following [39], we label the users with more than 80% helpful votes as normal users, and label users with less than 20% helpful votes as corrupted users. The Yelp data contains users and their comments on restaurants with true labels of the normal users and corrupted users.

We select 1,000 users and 1,000 items for Movielens; 1,400 users and 800 items for Amazon; 2,000 users and 2,000 items for Yelp. The ratios of corrupted users on these data are 10%, 3.5%, and

30.9%, respectively. We generate the preference and item vectors following [37, 21]. We first construct the binary feedback matrix through the users' ratings: if the rating is greater than 3, the feedback is 1; otherwise, the feedback is 0. Then we use SVD to decompose the extracted binary feedback matrix R u × m = θ SX T , where θ = ( θ i ) , i ∈ [ u ] and X = ( x j ) , j ∈ [ m ] , and select d = 50 dimensions. We have 10 clusters on Movielens and Amazon, and 20 clusters on Yelp. We use the same corruption mechanism as the synthetic data with T = 1 , 000 , 000 and k = 20 , 000 . Weconduct more experiments in different environments to show our algorithms' robustness in Appendix.G.

The recommendation results are shown in Fig.3(b)-(d). RCLUB-WCU outperforms all baselines. On the Amazon dataset, the percentage of corrupted users is lowest, RCLUB-WCU's advantages over baselines decrease because of the weakened corruption. The corrupted user detection results are provided in Tab.1. OCCUD's performance improves over time and is much better than GCUD. On the Movielens dataset, OCCUD achieves an AUC of 0.85; on the Amazon dataset, OCCUD

achieves an AUC of 0.84; and on the Yelp dataset, OCCUD achieves an AUC of 0.628. According to recent works on offline settings [18, 32], our results are relatively high.

7 Conclusion

In this paper, we are the first to propose the novel LOCUD problem, where there are many users with unknown preferences and unknown relations, and some corrupted users can occasionally perform disrupted actions to fool the agent. Hence, the agent not only needs to learn the unknown user preferences and relations robustly from potentially disrupted bandit feedback, balance the explorationexploitation trade-off to minimize regret, but also needs to detect the corrupted users over time. To robustly learn and leverage the unknown user preferences and relations from corrupted behaviors, we propose a novel bandit algorithm RCLUB-WCU. To detect the corrupted users in the online bandit setting, based on the learned user relations of RCLUB-WCU, we propose a novel detection algorithm OCCUD. We prove a regret upper bound for RCLUB-WCU, which matches the lower bound asymptotically in T up to logarithmic factors and matches the state-of-the-art results in degenerate cases. We also give a theoretical guarantee for the detection accuracy of OCCUD. Extensive experiments show that our proposed algorithms achieve superior performance over previous bandit algorithms and high corrupted user detection accuracy.

References

Yasin Abbasi-Yadkori, D´ avid P´ al, and Csaba Szepesv´ ari. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Charu C Aggarwal et al. Recommender systems , volume 1. Springer, 2016.

Ilija Bogunovic, Arpan Losalka, Andreas Krause, and Jonathan Scarlett. Stochastic linear bandits robust to adversarial attacks. In International Conference on Artificial Intelligence and Statistics , pages 991-999. PMLR, 2021.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011.

Qin Ding, Cho-Jui Hsieh, and James Sharpnack. Robust stochastic linear contextual bandits under adversarial attacks. In International Conference on Artificial Intelligence and Statistics , pages 7111-7123. PMLR, 2022.

Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S Yu. Enhancing graph neural network-based fraud detectors against camouflaged fraudsters. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 315-324, 2020.

Evrard Garcelon, Baptiste Roziere, Laurent Meunier, Jean Tarbouriech, Olivier Teytaud, Alessandro Lazaric, and Matteo Pirotta. Adversarial attacks on linear contextual bandits. Advances in Neural Information Processing Systems , 33:14362-14373, 2020.

Claudio Gentile, Shuai Li, and Giovanni Zappella. Online clustering of bandits. In International Conference on Machine Learning , pages 757-765. PMLR, 2014.

Anupam Gupta, Tomer Koren, and Kunal Talwar. Better algorithms for stochastic bandits with adversarial corruptions. In Conference on Learning Theory , pages 1562-1578. PMLR, 2019.

Mohammad Hajiesmaili, Mohammad Sadegh Talebi, John Lui, Wing Shing Wong, et al. Adversarial bandits with corruptions: Regret lower bound and no-regret algorithm. Advances in Neural Information Processing Systems , 33:19943-19952, 2020.

F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) , 5(4):1-19, 2015.

Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. In Advances in Neural Information Processing Systems (2022) , 2022.

Mengda Huang, Yang Liu, Xiang Ao, Kuan Li, Jianfeng Chi, Jinghua Feng, Hao Yang, and Qing He. Auc-oriented graph neural network for fraud detection. In Proceedings of the ACM Web Conference 2022 , pages 1311-1321, 2022.

Kwang-Sung Jun, Lihong Li, Yuzhe Ma, and Jerry Zhu. Adversarial attacks on stochastic bandits. Advances in neural information processing systems , 31, 2018.

Pushmeet Kohli, Mahyar Salek, and Greg Stoddard. A fast bandit algorithm for recommendation to users with heterogenous tastes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 27, pages 1135-1141, 2013.

Fang Kong, Canzhe Zhao, and Shuai Li. Best-of-three-worlds analysis for linear bandits with follow-the-regularized-leader algorithm. arXiv preprint arXiv:2303.06825 , 2023.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Qiutong Li, Yanshen He, Cong Xu, Feng Wu, Jianliang Gao, and Zhao Li. Dual-augment graph neural network for fraud detection. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 4188-4192, 2022.

Shuai Li and Shengyu Zhang. Online clustering of contextual cascading bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. Collaborative filtering bandits. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval , pages 539-548, 2016.

Shuai Li, Wei Chen, and Kwong-Sak Leung. Improved algorithm on online clustering of bandits. arXiv preprint arXiv:1902.09162 , 2019.

Yingkai Li, Edmund Y Lou, and Liren Shan. Stochastic linear optimization with adversarial corruption. arXiv preprint arXiv:1909.02109 , 2019.

Fang Liu and Ness Shroff. Data poisoning attacks on stochastic bandits. In International Conference on Machine Learning , pages 4042-4050. PMLR, 2019.

Shenghua Liu, Bryan Hooi, and Christos Faloutsos. Holoscope: Topology-and-spike aware fraud detection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , pages 1539-1548, 2017.

Xutong Liu, Haoru Zhao, Tong Yu, Shuai Li, and John CS Lui. Federated online clustering of bandits. In Uncertainty in Artificial Intelligence , pages 1221-1231. PMLR, 2022.

Xutong Liu, Jinhang Zuo, Siwei Wang, Carlee Joe-Wong, John Lui, and Wei Chen. Batch-size independent regret bounds for combinatorial semi-bandits with probabilistically triggered arms or independent arms. Advances in Neural Information Processing Systems , 35:14904-14916, 2022.

Xutong Liu, Jinhang Zuo, Siwei Wang, John CS Lui, Mohammad Hajiesmaili, Adam Wierman, and Wei Chen. Contextual combinatorial bandits with probabilistically triggered arms. In International Conference on Machine Learning , pages 22559-22593. PMLR, 2023.

Yang Liu, Xiang Ao, Zidi Qin, Jianfeng Chi, Jinghua Feng, Hao Yang, and Qing He. Pick and choose: a gnn-based imbalanced learning approach for fraud detection. In Proceedings of the Web Conference 2021 , pages 3168-3177, 2021.

Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 114-122, 2018.

Yuzhe Ma, Kwang-Sung Jun, Lihong Li, and Xiaojin Zhu. Data poisoning attacks in contextual bandits. In International Conference on Decision and Game Theory for Security , pages 186204. Springer, 2018.

Julian John McAuley and Jure Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd international conference on World Wide Web , pages 897-908, 2013.

Zidi Qin, Yang Liu, Qing He, and Xiang Ao. Explainable graph-based fraud detection via neural meta-graph search. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 4414-4418, 2022.

Shebuti Rayana and Leman Akoglu. Collective opinion spam detection: Bridging review networks and metadata. In Proceedings of the 21th acm sigkdd international conference on knowledge discovery and data mining , pages 985-994, 2015.

Daixin Wang, Jianbin Lin, Peng Cui, Quanhui Jia, Zhen Wang, Yanming Fang, Quan Yu, Jun Zhou, Shuang Yang, and Yuan Qi. A semi-supervised graph attentive network for financial fraud detection. In 2019 IEEE International Conference on Data Mining (ICDM) , pages 598607. IEEE, 2019.

Zhiyong Wang, Xutong Liu, Shuai Li, and John CS Lui. Efficient explorative key-term selection strategies for conversational contextual bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10288-10295, 2023.

Zhiyong Wang, Jize Xie, Xutong Liu, Shuai Li, and John Lui. Online clustering of bandits with misspecified user models. arXiv preprint arXiv:2310.02717 , 2023.

Junda Wu, Canzhe Zhao, Tong Yu, Jingyang Li, and Shuai Li. Clustering of conversational bandits for user preference learning and elicitation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 2129-2139, 2021.

Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in a collaborative environment. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval , pages 529-538, 2016.

Ge Zhang, Jia Wu, Jian Yang, Amin Beheshti, Shan Xue, Chuan Zhou, and Quan Z Sheng. Fraudre: fraud detection dual-resistant to graph inconsistency and imbalance. In 2021 IEEE International Conference on Data Mining (ICDM) , pages 867-876. IEEE, 2021.

Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Linear contextual bandits with adversarial corruptions. arXiv preprint arXiv:2110.12615 , 2021.

Jinhang Zuo, Zhiyao Zhang, Zhiyong Wang, Shuai Li, Mohammad Hajiesmaili, and Adam Wierman. Adversarial attacks on online learning to rank with click feedback. arXiv preprint arXiv:2305.17071 , 2023.

Appendix

A Proof of Lemma 1

We first prove the following result:

With probability at least 1 -δ for some δ ∈ (0 , 1) , at any t ∈ [ T ] :

where β ( T i,t , δ u ) ≜ √ 2 log( u δ ) + d log(1 + T i,t λd ) + λ + αC

where we denote M ′ i,t = M i,t + λ I , and the above equations hold by definition. Therefore, we have

We then bound the three terms in Eq.(9) one by one. For the first term:

where we use the Cauchy-Schwarz inequality, the inequality for the operator norm of matrices, and the fact that λ min ( M ′ i,t ) ≥ λ .

For the second term in Eq.(9), we have

where Eq.(11) follows by the Cauchy-Schwarz inequality and the inequality for the operator norm of matrices, and Eq.(12) follows by the Courant-Fischer theorem.

Let ˜ x s ≜ √ w i s ,s x a s , ˜ η s ≜ √ w i s ,s η s , then we have: ∥ ˜ x s ∥ 2 ≤ ∥ ∥ √ w i s ,s ∥ ∥ 2 ∥ x a s ∥ 2 ≤ 1 , ˜ η s is still 1-sub-gaussian (since η s is 1-sub-gaussian and √ w i s ,s ≤ 1 ), M ′ i,t = λ I + ∑ s ∈ [ t ] i s = i ˜ x s ˜ x ⊤ s , and the nominator in Eq.(12) becomes ∥ ∥ ∥ ∑ s ∈ [ t ] i s = i ˜ x s ˜ η s ∥ ∥ ∥ M ′-1 i,t . Then, following Theorem 1 in [1] and by union bound, with probability at least 1 -δ for some δ ∈ (0 , 1) , for any i ∈ U , we have:

where det ( · ) denotes the determinant of matrix arguement, Eq.(13) is because det ( M ′ i,t ) ≤ ( trace ( λ I + ∑ s ∈ [ t ] i s = i w i s ,s x as x ⊤ as ) d ) d ≤ ( λd + T i,t d ) d , and det ( λ I ) = λ d .

For the third term in Eq.(9), we have

where Eq.(14) follows by the Cauchy-Schwarz inequality and the inequality for the operator norm of matrices, Eq.(15) follows by the Courant-Fischer theorem, and Eq.(16) is because by definition w i,s ≤ α ∥ x as ∥ M ′-1 i,s ≤ α ∥ x as ∥ M ′-1 i,t (since M ′ i,t ⪰ M ′ i,s , M ′-1 i,s ⪰ M ′-1 i,t , ∥ x a s ∥ M ′-1 i,s ≥ ∥ x a s ∥ M ′-1 i,t ), ∑ T t =1 | c t | ≤ C .

Combining the above bounds of these three terms, we can get that Eq.(8) holds. We then prove the following technical lemma.

Lemma 6. Under Assumption 3, at any time t , for any fixed unit vector θ ∈ R d

where K is the upper bound of |A t | for any t .

Proof. The proof of this lemma mainly follows the proof of Claim 1 in [8], but with more careful analysis, since their assumption on the arm generation distribution is more stringent than our Assumption 3 by putting more restrictions on the variance upper bound σ 2 (specifically, they require σ 2 ≤ λ 2 8 log(4 K ) ).

Denote the feasible arms at round t by A t = { x t, 1 , x t, 2 , . . . , x t, |A t | } . Consider the corresponding i.i.d. random variables θ i = ( θ ⊤ x t,i ) 2 -E t [( θ ⊤ x t,i ) 2 | |A t | ] , i = 1 , 2 , . . . , |A t | . By Assumption 3, θ i s are sub-Gaussian random variables with variance bounded by σ 2 . Therefore, for any α > 0 and any i ∈ [ |A t | ] , we have:

where we use P t ( · ) to be the shorthand for the conditional probability P ( ·| ( i 1 , A 1 , r 1 ) , . . . , ( i t -1 , A t -1 , r t -1 ) , i t ) .

By Assumption 3, we can also get that E t [( θ ⊤ x t,i ) 2 | |A t | = E t [ θ ⊤ x t,i x ⊤ t,i θ | |A t | ] ≥ λ min ( E x ∼ ρ [ xx ⊤ ]) ≥ λ x . With these inequalities above, we can get

Therefore, we can get

Note that w i,s = min { 1 , α ∥ x as ∥ M ′-1 i,t } , and we have

Since α √ λ < 1 typically holds, we have w i,s ≥ α √ λ .

Then, with the item regularity assumption stated in Assumption 3, the technical Lemma 6, together with Lemma 7 in [19], with probability at least 1 -δ , for a particular user i , at any t such that T i,t ≥ 16 ˜ λ 2 x log( 8 d ˜ λ 2 x δ ) , we have:

With this result, together with Eq.(8), we can get that for any t such that T i,t ≥ 16 ˜ λ 2 x log( 8 d ˜ λ 2 x δ ) , with probability at least 1 -δ for some δ ∈ (0 , 1) , ∀ i ∈ U , we have:

Then, we want to find a sufficient time T i,t for a fixed user i such that

To do this, with Eq.(19), we can get it by letting

Then it is sufficient to get Eq.(26) if the following holds

Following Lemma 9 in [19], we can get the following sufficient condition for Eq.(30):

Then, since typically u δ > 288 γ 2 α √ λ ˜ λ x , we can get the following sufficient condition for Eq.(29) and Eq.(31)

Together with Eq.(24), Eq.(25), and the condition for Eq.(18) we can get the following sufficient condition for Eq.(20) to hold

Then, with Assumption 2 on the uniform arrival of users, following Lemma 8 in [19], and by union bound, we can get that with probability at least 1 -δ , for all

For Eq.(21), we can get

For Eq.(22), we can get

For Eq.(23), we have

For Eq.(27), we can get

For Eq.(28), we can get

Eq.(32) holds for all i ∈ U , and therefore Eq.(20) holds for all i ∈ U . With this, we can show that RCLUB-WCU will cluster all the users correctly after T 0 . First, if RCLUB-WCU deletes the edge ( i, l ) , then user i and user j belong to different ground-truth clusters, i.e., ∥ θ i -θ l ∥ 2 > 0 . This is because by the deletion rule of the algorithm, the concentration bound, and triangle inequality, ∥ θ i -θ l ∥ 2 = ∥ ∥ ∥ θ j ( i ) -θ j ( l ) ∥ ∥ ∥ 2 ≥ ∥ ∥ ∥ ˆ θ i,t -ˆ θ l,t ∥ ∥ ∥ 2 -∥ ∥ ∥ θ j ( l ) -θ l,t ∥ ∥ ∥ 2 -∥ ∥ ∥ θ j ( i ) -θ i,t ∥ ∥ ∥ 2 > 0 . Second, we show that if ∥ θ i -θ l ∥ ≥ γ , RCLUB-WCU will delete the edge ( i, l ) . This is because if ∥ θ i -θ l ∥ ≥ γ , then by the triangle inequality, and ∥ ∥ ∥ ˆ θ i,t -θ j ( i ) ∥ ∥ ∥ 2 < γ 4 , ∥ ∥ ∥ ˆ θ l,t -θ j ( l ) ∥ ∥ ∥ 2 < γ 4 , θ i = θ j ( i ) , θ l = θ j ( l ) , we have ∥ ∥ ∥ ˆ θ i,t -ˆ θ l,t ∥ ∥ ∥ 2 ≥ ∥ θ i -θ l ∥ -∥ ∥ ∥ ˆ θ i,t -θ j ( i ) ∥ ∥ ∥ 2 -∥ ∥ ∥ ˆ θ l,t -θ j ( l ) ∥ ∥ ∥ 2 > γ -γ 4 -γ 4 = γ 2 > √ λ + √ 2 log( u δ )+ d log(1+ T i,t λd ) √ λ +2 ˜ λ x T i,t + √ λ + √ 2 log( u δ )+ d log(1+ T l,t λd ) √ λ +2 ˜ λ x T l,t , which will trigger the deletion condition Line 10 in Algo.1.

B Proof of Lemma 2

After T 0 , if the clustering structure is correct, i.e. , V t = V j ( i t ) , then we have

where we denote M ′ V t ,t -1 = M V t ,t -1 + λ I , and Eq.(35) is because V t = V j ( i t ) thus θ i s = θ i t , ∀ i s ∈ V t .

Therefore, we have

where Eq.(36) is by Cauchy-Schwarz inequality, matrix operator inequality, and ∣ ∣ ∣ x ⊤ a M ′-1 V t ,t -1 θ i t ∣ ∣ ∣ ≤ λ ∥ ∥ ∥ M ′-1 2 V t ,t -1 ∥ ∥ ∥ 2 ∥ θ i t ∥ 2 = λ 1 √ λ min ( M Vt ,t -1 ) ∥ θ i t ∥ 2 ≤ √ λ since λ min ( M V t ,t -1 ) ≥ λ and ∥ θ i t ∥ 2 ≤ 1 .

Let ˜ x s ≜ √ w i s ,s x a s , ˜ η s ≜ √ w i s ,s η s , then we have: ∥ ˜ x s ∥ 2 ≤ ∥ ∥ √ w i s ,s ∥ ∥ 2 ∥ x a s ∥ 2 ≤ 1 , ˜ η s is still 1-sub-gaussian (since η s is 1-sub-gaussian and √ w i s ,s ≤ 1 ), M ′ i,t = λ I + ∑ s ∈ [ t ] i s = i ˜ x s ˜ x ⊤ s ,

and ∥ ∥ ∥ ∑ s ∈ [ t -1] i s ∈ Vt w i s ,s x a s η s ∥ ∥ ∥ M ′-1 Vt ,t -1 becomes ∥ ∥ ∥ ∑ s ∈ [ t ] i s = i ˜ x s ˜ η s ∥ ∥ ∥ M ′-1 Vt ,t -1 . Then, following Theorem 1 in [1], with probability at least 1 -δ for some δ ∈ (0 , 1) , we have:

And for ∥ ∥ ∥ ∑ s ∈ [ t -1] i s ∈ Vt w i s ,s x a s c s ∥ ∥ ∥ M ′-1 Vt ,t -1 , we have

where we use ∑ T t =1 | c t | ≤ C , w i s ,s ≤ α ∥ x as ∥ M ′-1 i s ,t -1 ≤ α ∥ x as ∥ M ′-1 Vt ,t -1 .

Plugging Eq.(38) and Eq.(37) into Eq.(36), together with Lemma 1, we can complete the proof of Lemma 2.

C Proof of Theorem 3

After T 0 , we define event

Then, with Lemma 1 and picking δ = 1 T , we have

Then it remains to bound I {E} R ( T ) . For the first T 0 rounds, we can upper bound the regret in the first T 0 rounds by T 0 . After T 0 , under event E and by Lemma 2, we have that with probability at least 1 -δ , for any x a :

Therefore, for the instantaneous regret R t at round t , with E , with probability at least 1 -δ , at ∀ t ≥ T 0 :

where the last inequality holds by the UCB arm selection strategy in Eq.(3) and Eq.(41).

Therefore, for I {E} R ( T ) :

Then it remains to bound E [ I {E} ∑ T t = T 0 +1 C a t ,t ] . For ∑ T t = T 0 +1 C a t ,t , we can distinguish it into two cases:

Then, we prove the following technical lemma. Lemma 7.

Proof.

∀ x ∈ [0 , 1] , we have x ≤ 2 log(1 + x ) . Therefore

Denote the rounds with w i t ,t = 1 as { ˜ t 1 , . . . , ˜ t l 1 } , and gram matrix ˜ G V ˜ t τ , ˜ t τ -1 ≜ λ I + ∑ s ∈ [ τ ] i s ∈ V ˜ t τ x a ˜ t s x ⊤ a ˜ t s ; denote the rounds with w i t ,t < 1 as { t ′ 1 , . . . , t ′ l 2 } , gram matrix G ′ V t ′ τ ,t ′ τ -1 ≜ λ I + ∑ s ∈ [ τ ] i s ∈ V t ′ τ w i t ′ s ,t ′ s x a t ′ s x ⊤ a t ′ s .

Then we have

where Eq.(48) is because ˜ G -1 V ˜ t τ , ˜ t τ -1 ⪰ M -1 V ˜ t τ , ˜ t τ -1 in Eq.(49) we use Cauchy-Schwarz inequality, in Eq.(50) we use Lemma 7 and ∑ l 1 τ =1 I { i ˜ t τ ∈ V j } ≤ T V j ,T , in Eq.(51) we use Cauchy-Schwarz inequality and ∑ m j =1 T V j ,T = T .

For the second part in Eq.(44), Let x ′ a t ′ τ ≜ √ w i t ′ τ ,t ′ τ x a t ′ τ , then

where in Eq.(52) we use the definition of the weights, in Eq.(53) we use G ′-1 V t ′ τ ,t ′ τ -1 ⪰ M -1 V t ′ τ ,t ′ τ -1 , and Eq.(54) uses Lemma 7.

Thus we complete the proof of Theorem 3.

D Proof and Discussions of Theorem 4

Table 1 of the work [12] gives a lower bound for linear bandits with adversarial corruption for a single user. The lower bound of R ( T ) is given by: R ( T ) ≥ Ω( d √ T + dC ) . Therefore, suppose our problem with multiple users and m underlying clusters where the arrival times are T i for each cluster, then for any algorithms, even if they know the underlying clustering structure and keep m independent linear bandit algorithms to leverage the common information of clusters, the best they can get is R ( T ) ≥ dC + ∑ i ∈ [ m ] d √ T i . For a special case where T i = T m , ∀ i ∈ [ m ] , we can get

Recall that the regret upper bound of RCLUB-WCU shown in Theorem 3 is of O ( ( C √ d γ 2 ˜ λ x +

bound with respect to T up to logarithmic factors and with respect to C up to O ( m ) factors, showing the tightness of our theoretical results (where m are typically very small for real applications).

We conjecture that the gap for the m factor in the mC term of the lower bound is due to the strong assumption that cluster structures are known to prove our lower bound, and whether there exists a tighter lower bound will be left for future work.

E Proof of Theorem 5

We prove the theorem using the proof by contrapositive. Specifically, in Theorem 5, we need to prove that for any t ≥ T 0 , if the detection condition in Line 7 of Algo.2 for user i , then with probability at least 1 -5 δ , user i is indeed a corrupted user. By the proof by contrapositive, we can prove Theorem 5 by showing that: for any t ≥ T 0 , if user i is a normal user, then with probability at least 1 -5 δ , the detection condition in Line 7 of Algo.2 will not be satisfied for user i .

If the clustering structure is correct at t , then for any normal user i

where ˜ θ i,t is the non-robust estimation of the ground-truth θ i , and ˆ θ V i,t ,t -1 is the robust estimation of the inferred cluster V i,t for user i at round t . Since the clustering structure is correct at t , ˆ θ V i,t ,t -1 is the robust estimation of user i 's ground-truth cluster's preference vector θ j ( i ) = θ i at round t . We have

where we denote ˜ M ′ i,t ≜ λ I + ∑ s ∈ [ t ] i s = i x a s x ⊤ a s , and Eq.(57) is because since user i is normal, we have c s = 0 , ∀ s : i s = i .

Then, we have

where Eq.(58) follows by the Cauchy-Schwarz inequality and the inequality for the operator norm of matrices, and Eq.(59) follows by the Courant-Fischer theorem and the fact that λ min ( ˜ M ′ i,t ) ≥ λ . Following Theorem 1 in [1], for a fixed normal user i , with probability at least 1 -δ for some δ ∈ (0 , 1) we have:

where Eq.(60) is because det ( ˜ M ′ i,t ) ≤ ( trace ( λ I + ∑ s ∈ [ t ] i s = i x as x ⊤ as ) d ) d ≤ ( λd + T i,t d ) d , and det ( λ I ) = λ d .

Plugging this into Eq.(59), we can get

Therefore, we have

Let ˜ x s ≜ √ w i s ,s x a s , ˜ η s ≜ √ w i s ,s η s , then we have: ∥ ˜ x s ∥ 2 ≤ ∥ ∥ √ w i s ,s ∥ ∥ 2 ∥ x a s ∥ 2 ≤ 1 , ˜ η s is still 1-sub-gaussian (since η s is 1-sub-gaussian and √ w i s ,s ≤ 1 ), M V i,t ,t = λ I + ∑ s ∈ [ t ] i s ∈ V j ( i ) ˜ x s ˜ x ⊤ s , and

∥ ∥ ∥ ∥ ∑ s ∈ [ t ] i s ∈ V j ( i ) w i s ,s x a s η s ∥ ∥ ∥ ∥ M -1 V i,t ,t becomes ∥ ∥ ∥ ∥ ∑ s ∈ [ t ] i s ∈ V j ( i ) ˜ x s ˜ η s ∥ ∥ ∥ ∥ M -1 V i,t ,t . Then, following Theorem 1 in [1], with probability at least 1 -δ for some δ ∈ (0 , 1) , for a fixed normal user i , we have

where Eq.(60) is because det ( M V i,t ,t ) ≤ ( trace ( λ I + ∑ s ∈ [ t ] i s ∈ V j ( i ) x as x ⊤ as ) d ) d ≤ ( λd + T V i,t ,t d ) d , and det ( λ I ) = λ d .

where Eq.(67) is because w i s ,s ≤ α ∥ x as ∥ M ′-1 i s ,s ≤ α ∥ x as ∥ M ′-1 i s ,t ≤ α ∥ x as ∥ M -1 V i,t ,t (since M V i,t ,t ⪰ M ′ i s ,t ⪰ M ′ i s ,s , M ′-1 i s ,s ⪰ M ′-1 i s ,t ⪰ M -1 V i,t ,t , ∥ x a s ∥ M ′-1 i s ,s ≥ ∥ x a s ∥ M ′-1 i s ,t ≥ ∥ x a s ∥ M -1 V i,t ,t ), and ∑ s ∈ [ t ] | c s | ≤ C .

Therefore, we have

With Eq.(68), Eq.(61) and Eq.(56), together with Lemma 1, we have that for a normal user i , for any t ≥ T 0 , with probability at least 1 -5 δ for some δ ∈ (0 , 1 5 )

which is exactly the detection condition in Line 7 of Algo.2.

Therefore, by the proof by contrapositive, we complete the proof of Theorem 5.

F Description of Baselines

We compare RCLUB-WCU to the following five baselines for recommendations.

LinUCB[17]: A state-of-the-art bandit approach for a single user without corruption.

LinUCB-Ind: Use a separate LinUCB for each user.

CW-OFUL[12]: A state-of-the-art bandit approach for single user with corruption.

CW-OFUL-Ind: Use a separate CW-OFUL for each user.

CLUB[8]: A graph-based clustering of bandits approach for multiple users without corruption.

SCLUB[21]: A set-based clustering of bandits approach for multiple users without corruption.

G More Experiments

G.1 Different Corruption Levels

To see our algorithm's performance under different corruption levels, we conduct the experiments under different corruption levels for RCLUB-WCU, CLUB, and SCLUB on Amazon and Yelp datasets. Recall the corruption mechanism in Section 6.1, we set k as 1,000; 10,000; 100,000. The results are shown in Fig.4. All the algorithms' performance becomes worse when the corruption level increases. But RCLUB-WCU is much robust than the baselines.

1k

10k

100k



0

1

2

3

4

5

6



1e4

RCLUB-WCU

CLUB

SCLUB

(a) Amazon Corruption Level

1k

10k

100k



0

2

4

6

8



1e4

RCLUB-WCU

CLUB

SCLUB

(b) Yelp Corruption Level

5

10

20

50



0

1

2

3

4

5

6

7



1e4

RCLUB-WCU

CLUB

SCLUB

(a) Amazon Cluster Number

5

10

20

50



0.0

0.2

0.4

0.6

0.8

1.0



1e5

RCLUB-WCU

CLUB

SCLUB

(b) Yelp Cluster Number

G.2 Different Cluster numbers

Following [19], we test the performances of the cluster-based algorithms (RCLUB-WCU, CLUB, SCLUB) when the underlying cluster number changes. We set m as 5, 10, 20, and 50. The results are shown in Fig.5. All these algorithms' performances decrease when the cluster numbers increase, matching our theoretical results. The performances of CLUB and SCLUB decrease much faster than RCLUB-WCU, indicating that RCLUB-WCU is more robust when the underlying user cluster number changes.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an important online learning problem named LOCUD to learn and utilize unknown user relations from disrupted behaviors to speed up learning and identify the corrupted users in an online setting. Also, the authors propose a novel bandit algorithm RCLUB-WCU, and devise a novel online detection algorithm OCCUD based on RCLUB-WCU’s inferred user relations. Extensive experiments demonstrate that the proposed methods can achieve superior performance over previous bandit algorithms and high corrupted user detection accuracy.\n\nSTRENGTHS:\n1. The paper is scientifically sound.\n2. The clarity of the presentation is easy to follow.\n3. Extensive experiments of the proposed methods have superior performance than other baselines.\n\nWEAKNESSES:\n1. The introduction section of the paper lacks sufficient emphasis on the motivation behind the proposed methods. The authors should provide a more comprehensive analysis of the current issues and challenges in the relevant fields, clearly indicating how their research work addresses and improves upon these challenges. This will help readers better understand the significance and contributions of the proposed methodology.\n2. The abstract section should be more concise. It should effectively highlight the key innovations and improvements introduced by the proposed methodology.\n3. The paper provides limited discussion and summary of the relevant literature. To strengthen the research methodology, the authors should include a more extensive review of existing research methods, along with an analysis of their strengths and weaknesses. \n4. The experimental content is not enough to effectively prove the superiority of the method. It is suggested that the authors add more dimensional experiments and give a comprehensive analysis and explanation of the experimental results, thus making the conclusions in the paper more convincing.\n5. The dataset used in the experiments is relatively small, which may limit the generalizability of the findings. It is recommended to supplement a large-scale real dataset for performance validation.\n\nQUESTIONS:\nProvided above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the following bandit setup.\n\nThere are $u$ users organised into $m\\ll u$ clusters. Each cluster has vector $\\theta$ attached to it. On step $t$ the learner deals with a user uniformly selected from the pool and picks an arm $a$. If this is a bona fide user, the learner gets average reward $x'_a\\theta$, where $x_a$ is the feature vector for the arm and $\\theta$ is the vector for the cluster the user belongs to. There is a number of corrupted users though, who give reward $x'_a\\theta + \\eta + c$, where $\\eta$ is noise and $c$ (a bounded quantity) is corruption.\n\nWe have a twofold problem of minimising the regret and identifying the corrupted users. The paper presents an algorithm based on a graph of connections between users built by observations of their behaviour. The regret upper bound is matched by a lower bound. It is shown that with high probability we identify the corrupted users correctly and after a while end up with correct clusters.\n\nSTRENGTHS:\nI think this is an interesting result and a strong guarantee. The explicit modelling of corrupted user behaviour may seem restrictive at first, but it covers many possibilities. The authors were very careful to relax all the requirements as much as possible (e.g., consider sub-Gaussian noise etc). The algorithm is intuitive.\n\nWEAKNESSES:\nNo obvious weaknesses. The appeal of the result may be limited for some NeurIPS participants.\n\nSome minor suggestions (not reflected in my evaluation of the paper):\n\n1. I do not think quantifiers should be used as in \n\nAt $\\forall t$, for any fixed unit vector $z$ ...\n\n(Assumption 3.) \n\nUsing \"every\" here will not blow the volume of the paper out of proportions, but will improve readability.\n\n2. Use $\\verb!\\left(...\\right)!$ in formulas like (7) to get larger brackets.\n\n3. I do not find the abbreviations used in the paper convenient and phrases such as\n\nCW-OFUL-Ind outperforms LinUCB-Ind because it considers the corruption, but worse than RCLUB-WCU\n\neasy to parse.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce an online learning problem called LOCUD (Learning and Online Corrupted Users Detection from bandit feedback) in which the aim is to detect a small fraction of the overall users with corrupt behaviors; corrupt users occasionally perform undesirable actions, but otherwise mimic normal user behavior, making them challenging to detect.\n\nThe paper then proposes a framework that leverages the relations between users to form semantic clusters, and uses the clusters to identify corrupt users. Specifically, the authors propose RCLUB-WCU (Robust CLUstering of Bandits With Corrupted Users) to progressively prune a fully connected graph of users to clusters of connected components based on user interactions and preferences. Then, OCCUD (Online Cluster-based Corrupted User Detection) estimates a robust and non-robust estimation of each user's preferences, and identifies a user as corrupt if the gap between the two estimates exceeds a carefully designed threshold. OCCUD is repeatedly invoked within RCLUB-WCU to continually prune and refine the relational structure amongst the users.\n\nExperiments one on synthetic dataset and three real-world datasets show the proposed approach is able to detect more corrupted users while achieving the least amount of regret over time than competing methods.\n\nSTRENGTHS:\n* The paper introduces a challenging but relevant problem of trying to identify corrupt users despite sporadic behavior in an online dynamic environment. This problem is especially relevant for sites like Amazon and Yelp which often contain users that exhibit corrupt behavior.\n\n* The proposed approach of leveraging relational information between users to more effectively detect corrupt users is intuitive, and the experimental results suggest its effectiveness.\n\n* Analyses and bounds related to regret are given for the RCLUB-WCU/OCCUD algorithm, with proofs provided in the Appendix.\n\n* The paper is generally well-written and follows a logical progression.\n\nWEAKNESSES:\n* Experiments are performed on a small number of small datasets, potentially limiting the generalizability of the proposed approach. Performing experiments with a wider range of larger datasets would significantly benefit the claims made in the paper.\n\n* No empirical runtime analysis is provided for the proposed approach or any of the competing methods. Runtime analysis can help practitioners decide what method is likely to work best for their particular problem.\n\n* Only one baseline method is compared to the proposed approach for the AUC results (Table 1). Where are the results for the other methods?\n\n* Additional experimental results including different corruption levels and number of clusters are provided in the Appendix, but those results (Figures 4 and 5) only shows regret, and only compares the proposed approach with two baseline methods. Can the authors provide results for the other baseline methods, as well as AUC results for these additional and potentially insightful experiments?\n\n* Minor clarity improvements:\n    * Use authors' names to cite previous work. \"The work [5] proposes...\" -> \"Ding et al. (2022) propose...\".\n    * The legends in Figure 5 make the subplots hard to read. Figure 5 is also not color-blind friendly, consider adding markers or using different line styles for different methods.\n\nQUESTIONS:\n* Why are the dataset sizes so small (e.g., 1,400 users and 800 items for Amazon)? What are the original sizes of the datasets used in the experiments?\n\n* Where are the error bars for Table 1, and Figures 4 and 5?\n\nLIMITATIONS:\nNo, the authors have not addressed the limitations or the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research paper introduces an innovative method for learning and utilizing unknown user relations from disrupted behaviors to enhance the learning process and identify corrupted users in an online setting. To achieve this, a new bandit algorithm (RCLUB-WCU) is proposed, along with an online detection algorithm that leverages user relations inferred by RCLUB-WCU. The paper also presents a regret upper bound for RCLUB-WCU, which closely matches the lower bound with respect to T (the number of rounds) up to logarithmic factors and performs well even in degenerate cases. The experiments conducted on synthetic and real-world datasets demonstrate significant improvements in performance compared to previous bandit algorithms\n\nSTRENGTHS:\n1. The paper presents a novel application to learn unknown user relations in their preferences from potentially corrupted feedback. At the same time, the paper shows how to leverage the learned relations to speed up learning as well as adaptively detect the corrupted users online from bandit feedback. Overall I think this paper makes a significant contribution to the literature on online learning from corrupted feedback as well as detection of adversarial users in multi-user online learning setting. \n\n2. Experiments on synthetic and real-world datasets clearly indicate  lower reward regret of proposed approach in comparison to five other baselines from the past literature on online clustering of bandits. At the same time, the algorithm is able to identify corrupted users with higher accuracy than a simple baseline that directly compares the robust-estimators of preference vectors of a user and its corresponding cluster.\n\n3. The authors back up their results with theoretical analysis and guarantees on the performance of the proposed algorithm.\n\nWEAKNESSES:\n1. It seems to me that the performance of proposed approach could potentially be sensitive to the nature of underlying user relations in their preferences and the tightness of the detected clusters. For example, to my understanding, this algorithm might not work well if there are too many tight clusters. However, if one chooses too big clusters, then the solution might compromise on personalization as users in a loose cluster might not be represented accurately by a common preference vector defined for the cluster. The paper does not provide any discussion on the impact of cluster sizes on the algorithm. \n\n2. The approach requires multiple parameters to be specified. (e.g. regularization parameter, confidence radius parameter, threshold parameter, edge detection parameter). There is no discussion provided on the sensitivity of the results to the choices of these parameters (in the main draft at least).\n\n3. The results in Table 1 on detection of corrupted users are not very clear. To begin with, it's not clear what do numbers in the table indicate? I assume they indicate recall of true corrupted users. If so, where are the precision numbers? Where are F1 scores given that thre is class imbalance? Authors need to clearly indicate what they are measuring and also provide an explanation in case they aren't measuring both precision and recall. Also it would be nice to have comparison with multiple baselines. Further, authors could potentially enrich the results with other variants of the baseline, e.g. another baseline variant could simply compare the non-robust estimators of cluster and user. (I believe the current baseline compares the robust estimators of a user and its cluster).\n\nQUESTIONS:\n1. Page 9, lines 341-345: Authors claim that the performance of the proposed algorithm improves with proportion of corrupted users in the dataset. Seems like this argument is made based on the three datapoints (from three real-world datasets). This argument could be potentially strengthened by evaluating all algorithms while varying the number of corrupted users in one of the datasets.\n\n2. Page 6: Line 220: It says that the threshold is carefully designed to handle the estimation uncertainty... How difficult/easy it is to tune this threshold? How critical this threshold is to the performance of the approach?\n\n3. Pages 4-5, Lines 188-204: It seems like the subscript `t` is used to denote both timestamp (bandit round) as well as the cluster? Also I see subscript `s` to indicate the bandit iteration which should ideally be denoted by t ?\n\n4. In Equation (2), are you considering data from all t -1 previous rounds? Or are you using iterative update and only include the data from previous round, i.e. (t -1)^{th} round?\n\nLIMITATIONS:\nAuthors should cover any other limitations of this work in addition to potential limitations I highlighted above:", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}