Graph Neural Bandits
Authors: Yunzhe Qi
Yikun Ban
Jingrui He
Venue: Submitted to ICLR 2023 (2023)
Keywords: Contextual Bandits
Graph Neural Networks

Abstract: Contextual bandits aim to choose the optimal arm with the highest reward out of a set of candidates based on their contextual information, and various bandit algorithms have been applied to personalized recommendation due to their ability of solving the exploitation-exploration dilemma. Motivated by online recommendation scenarios, in this paper, we propose a framework named Graph Neural Bandits (GNB) to leverage the collaborative nature among users empowered by graph neural networks (GNNs). Instead of estimating rigid user clusters, we model the "fine-grained'' collaborative effects through estimated user graphs in terms of exploitation and exploration individually. Then, to refine the recommendation strategy, we utilize separate GNN-based models on estimated user graphs for exploitation and adaptive exploration. Theoretical analysis and experimental results on multiple real data sets in comparison with state-of-the-art baselines are provided to demonstrate the effectiveness of our proposed framework.

Under review as a conference paper at ICLR 2023
GRAPH NEURAL BANDITS
Anonymous authors Paper under double-blind review
ABSTRACT
Contextual bandits aim choose the optimal arm with the highest reward out of set of candidates based on their contextual information, and various bandit algorithms have been applied to personalized recommendation due to their ability of solving the -oooitation-eplloatttoonn dilemma. Motivated by online recommen- dation scenarios, in this paper, we propose framework named Graph Neural Bandits (GNB) to leverage the collaborative nature among users empowered by graph neural networks (GNNs). Instead of estimating rigid user clusters, we model the "fine-grained" collaborative effects through estimated user graphs in terms exploitation and exploration individually. Then, to refine the recommendation strategy, utilize separate GNN- models on estimated user graphs for ex- ploitation and adaptive exploration. Theoretical analysis and experimental results multiple real data sets in comparison with state-of-the-an baselines are provided to demonstrate the effectiveness of our proposed framework.
INTRODUCTION
Contextual bandits are specific type of multi-armed bandit problem where the additional contextual information (contexts) related to arms are available at each round, and the learner intends to refine its selection strategy based the received arm contexts and rewards. Various contextual bandit algorithms have been applied in real-world recommendation tasks, such as online content recom- mendation and advertising (Li al. 2010 Wu et al. 2016), and clinical trials (Durand al. 2018 Villar et al. 2015). Meanwhile, collaborative effects among users provide us the opportunity to design better recommender strategies, since the target user's preference can be inferred based on other similar users. Such effects have been studied by many bandit works (Gentile et al.) 2014. Li jtt al. 2019; Gentile et al.) 2017 et al.) 2016 Ban He 202T). Different from the conventional collaborative filtering methods (He et al. 2017 Wang et al. 2019). bandit-based approaches focus on more dynamic environments (such as news, short-video platform) and the exploiaation-explrraiin dilemma inherently existed in the decisions of recommendation.
Existing works for clustering of bandits (Gentile et al., 2014 Li et al. 2019: Gentile et al. 201
et al. 2016; Ban & He 2021 Ban et al. 2022a) have been proposed to model the user correlations (collaborative effects) by clustering users into rigid groups, and assigning each formed group with an estimator to learn the assumed reward functions combined with an Upper Confidence Bound (UCB) strategy for exploration. However, these works only consider the "coarse-grained user correlations. To be specific, they assume that users from the same group would share identical preferences, i.e., the users from the same group are compelled to make equal contributions to the final decision (arm selection) with regard to the target user. Such formulation of user correlations ("coarse- grained" collaborative effects), evidently fails to comply with real-world application scenarios, since users within the same group tend to have similar but subtly different preferences instead of sharing completely identical tastes. Therefore, given a target user, it is more practical to assume that the rest. of the users would impose different levels of (collaborative) effects on this user.
Motivated by aforementioned limitations of existing works, in this paper, we propose novel framework, named Graph Neural Bandits (GNB), to formulate the "fine-grained" collaborative effects, where the correlation of each user pair is preserved by user graphs. Given target user, other users are allowed to make different contributions to the final decision based on the strength of their correlation the target user, which therefore corresponds to the "fine- grained" collaborative effects. In particular, in GNB, we propose a novel approach to construct two kinds of user graphs with distinct purposes, called "user exploitation graphs" and "user exploration graphs". Then, apply

---

Under review as a conference paper at ICLR 2023
two separate graph neural network (GNN) models these two kinds of user graphs, to incorporate the collaborative effects for both exploitation and exploration purposes in the final decision decision-making makinng process. Our main contributions can be summarized as follows:
1. Different from existing works that only formulate the "coarse-grained collaborative effects by neglecting the divergence within user groups, we introduce a new problem setting to model the "fine- giane-gained" user collaborative effects via user graphs. In setting, the pair-wise user correlations are preserved to contribute differently to the decision-making.
2. We propose framework named GNB, which has the novel ways to build two kinds of user graphs with two different purposes, i.e., exploitation and adaptive exploration, respectively. Then, GNB utilizes GNN-based models for a refined arm selection strategy by leveraging the user correlations encoded in these two kinds of user graphs.
3. With standard assumptions, we provide the theoretical analysis showing that GNB can achieve the regret upper bound of complexity rounds and n is the number of users. This bound is sharper than the existing related works. experimente
(TVologg))) where 7 is the number of
conducted 4. Extensive experiments comparing GNB with nine state-of- the-art algorithms
on various real data sets, which demonstrate the effectiveness of our proposed method.
we provide the details of our proposed After introducing the problem definition Section [
framework in 3 Then, present the theoretical analysis in Section and the experiments in Section [ Finally, we conclude the paper in Section 6 Due to page limit, we will leave the review
of related works the Section A in the Appendix.
GRAPH NEURAL BANDITS: PROBLEM DEFINITION AND NOTATION
Suppose there are total of n users with the user set U {1. n). At each time step t E [T]. the learner will receive a user un E U to serve, along with a set of candidate arms X₂ = { he(() for selection. The cardinality of this arm set |. = a, and each arm described by d-dimensional context vector D4, E Rd with || = 1. Mcanwhile, each arm Ti,t E X, is associated with reward ri,-- As the user correlation is one important factor in determining the reward, we define the following reward function:
Ti. = h( Ue, (()).)) + Eit
where h(-) is the unknown reward mapping function, and e,,r stands for some zero-mean noise such that E[[r,.] h( Ue, 9 ((.).)). Here, we have g(I),+ = (4, F. w(l).') being the unknown user graph induced by arm Ei,t, which encodes the ffine-grained user correlations in terms of
expected rewards. In graph g()),, each user u E U corresponds to a graph node; meanwhile,
E = {e(u, u') va, refers to the set of edges, and the set w:!!:+ = {wift (1)* (u, u') stores the weights for each edge from E. Under real-world application scenarios, users sharing the same preference for certain arms (e.g., sports news) may have distinct tastes over other arms (e.g., political (1),,
news). Thus, allow each arm T X to induce different user collaborations Gi
Then, motivated by various real applications (e.g., online recommendation with normalized ratings), we consider ri.t to be bounded Ti. E 00, 1] in this paper. which is standard in existing works (e.g.. Gentile et al. 20144 2017) y Ban & He 2021 : Ban et al. (2022a)). Note that as long as Ti, E [0, 1], do not impose any distribution assumption (e.g., sub- Gaussian distribution) on noise term 6,,--
Comparison with Existing Problem Definitions. The problem definition of existing user clustering works (e.g.. Gentile et al. (0144): Li et al. (2019): Gentile al. (2017); (Ban & He (202): Ban et al. (2022a)) only can formulate "coarse-grained user correlations. In their settings, given a user group N C U. all the users in A are forced share the same reward function given an arm 24,,, i.c.. Blrie u, xx,1] = (w( Vu E N. In contrast, our definition of the reward function enables us to model the pair-wise fine-grained user correlations by introducing another two important factors " and G With our formulation, each user here is allowed to produce different rewards facing the same arm, i.e., E(ri, | u. x.,]] = h(x,,,, G
(1)).)) Vu E N. Here, with different users u,
the corresponding expected reward h( u, G can be different. Therefore, our definition of the reward function is more generic, and it can also readily generalize existing user clustering algorithms (with "coarse-grained" user correlations) by allowing each single user group to form an isolated sub-graph in g()),* with no connections across different sub-graphs.
2

---

Under review as a conference paper at ICLR 2023
To bridge user collaborative effects with user preferences (rewards), consider the following constrain for the reward function in Eq. The intuition is that for any two users with comparable user correlations, they would share similar tastes over the items with high probability. For arm a we consider the difference of expected rewards between any two users u, u/ E U to be governed by
11 u, -/- h( u (00 S VIG (M [u. 1.
(2)
where G [u, :| stands for the adjacency matrix row of G that corresponds to user (node) u, and IR R + R denotes an unknown mapping function. The reward function definition (Eq.
and the constraint (Eq. 2) motivate us to design the GNB framework, to be introduced in Section 3
Since the user graph G.P.'* in Eq. reflects user correlations in terms of the expected reward, is exactly referring to the user exploitation correlation, where users with high correlations tend to
have similar expected rewards (Eq. Then, proceed give the formulation of G below.
Definition 1 (User Correlation for Exploitation). round t, for any two users u, u' u. their (1):* exploitation correlation score Wii (u, u') w.r.. candidate arm Diit N2 is defined as
(1) (u, ('), ( T (E(ri, v.aa,, Biri,lud, w.,.))
where El[ra|u, 2r,c), E [a] the expected reward terms of the user-arm pair (u, 2r,)). Given two users u, E U. the function (((); R x R +> I maps from their expected rewards E[ri, u, æi,el to their user exploitation score wit (u, w'). (1),*
Given an arm Fi, E Xr the user correlation for exploitation measures the user preference (i.e., expected reward) correlation between two users u, u' U, and the corresponding exploitation score (1).* (u, u') refers to the edge weight between these two users (nodes) u' in exploitation graph Wie g..)."" In this paper, we consider the mapping functions (1)) as the prior knowledge, and it can be functions such as the radial basis function (RBF) kernel or normalized absolute difference in practice.
Modeling User Exploration Correlations with the User Exploration Graph (!2),* In order to deal with the pnpllratioo--eploitaaiinn dilemma, for each candidate arm Di, Xr we propose to formulate the user exploration graph g(?,** (44, F, w(?):t) " in order to model the user correlations in terms of the uncertainty of the reward estimation model, which are formulated by the set of edge weights w {w (2).* u') ( vv.. Inspired by Ban et al. (2022b), before defining the second kind of user correlation (i.e., user exploration correlation), we first introduce the definition of expected potential gain for reward estimation, which measures the prediction uncertainty of reward estimators. Note that our formulation distinct from Ban al. (20226), since they only focus on the single-bandit setting with user collaborations, and all the users will be treated identically.
Definition 2 (Expected Potential Gain). Given user u E U at time step t, given candidate arm 2i,t E X,, [a] and a reward estimation function ful.) corresponding to user u, the expected potential gain for the reward estimation (x( is defined as El (M, Fi,e] fu(wi,))
Here, the potential gain for reward estimation essentially formulates the uncertainty of model fu(-) by measuring the difference between the expected reward E/ri,//u, æi,e] and the prediction fa( (x...) Next, we proceed to introduce the second kind of user correlation, i.e., user exploration correlation.
Definition 3 (User Correlation for Exploration). In round t, given two users u, ul E ll and an arm Di,t Xr. their underlying exploration correlation score watt
(2)2. (u, is
(,,2),* (u, u') (2)) (E(rs,cu,, æ.,1] - -f,(( Elra,,,'' ai.) fwe
with Efrialu, Dite] - fu(ax,)), E [a] being the potential gain for the user-arm pair (u, F.,)). Here, fa()) is the reward estimation function specified to user u, and 4(22 : B x R + R is the mapping from user potential gains Elr,,, æv.1] (xx,)) to their exploration correlation score.
For the æ. and two users u, u' U. the user exploration correlation score w., (u, u') refers to the correlation of prediction uncertainty between two user-specific functions fa()) and fw (.). Then,
(2), (u,
3

---

Under review as a conference paper at ICLR 2023
Parameter Update 6D
True Rdge Weight to EP motte
(E/ Elr (g)=
Exploitation User Graph Esi.
Si
Reward Rr
Explottation GNN (a
Select Arm argnaxy,eet (ir B(t))
User-Arm Pair (ug)
Feed to
Get San Feed
Exploration User Graph Est.
Potential Gain Est. hn
S
Thus Edge Wasget Eahinahe
[[ My. fu. (Fr)), Efrul xul 1- (x)()
Exploration GNN (eveE)
1. User Graph Estimation
2. Reward Potential Gain Estimation
3. Arm Selection
Figure Workflow of the proposed Graph Neural Bandits (GNB) framework.
(2).x the exploration score Wist (u, u²) will be considered as the edge weight between these two nodes (users) u, uf in the true user exploration graph g(2),* Analogous to previous (1),, we also consider the mapping functions vy(2) is given as the prior knowledge. Intuitively, when the exploration score wit (u, u/) is high, we can apply similar exploration strategies for both users u, u'. For example, given arm iitt if the reward estimation error (i.e., prediction uncertainty) is large for both u and u', we may want to explore these two user-arm pairs (u, at,t), (w', aii,)) more for additional knowledge.
Learning Objective. For the received user ue in round t, the learner expected recommend an arm Ie E Xg (with reward rt) order to minimize the cumulative pseudo-regret R(T) = EE (rt
r:)] where rī the reward for the optimal arm E[r [u,, X,] max (a
We, g())*))
Notation, Denoting Tu, S [d] as the collection of time steps that user E 2 is served up round we use Pa.e = {(x,, rr FreT... to represent the collection of received arm-reward pairs associated with user u, and Tu,t = [Tu,t] refers to the number of rounds that user has been served. Here, I, E Ar. Pr E R separately refer to the chosen arm and actual received reward in round T E Tu,t- Similarly, we use P.
to denote all the past records (i.e., reward pairs), up to round For any graph G, we denote A Roxx as its adjacency matrix (with added self-loops), and D _R as its degree matrix. Then, we will introduce proposed solution, the GNB framework.
GRAPH NEURAL BANDITS: PROPOSED FRAMEWORK
The workflow of our proposed GNB framework is illustrated by Figure [ and consists of four major
components: (1) estimating the exploitation graph g(1),,, denoted by ((2), and user exploration graph g(2),,, denoted by (2) model the correlations in terms of exploitation and exploration respectively; (2) applying GNN models fg / (.) on the estimated user graphs (()) and g(2), to collaboratively derive the estimated reward for exploitation, and potential gain for exploration; (3)
selecting the arm 2, based on estimated reward and potential gain; and (4) training parameters for GNN models and neural networks with gradient descent (GD).
3. USER GRAPH ESTIMATION
USER NETWORKS
(1),+ Based on the definition of unknown true user graphs g _g(2)* w.r.t. arm Ri, E X' (Definitions
we proceed to derive their estimations (( _ [a] with individual user networks f((
thus model the user behaviors under exploitation setting and the exploration setting separately. Due to page limit, pseudo-code summarizing the workflow is presented in Alg. 2 in Section of the Appendix.
fa E U. With these two kinds of estimated user graphs G and ((2)
User Exploitation Network ( For each user E U4 we propose to apply an exploitation network ((2) (.) to learn user u's preference for arm F;,r, i.e., EE[r,rlu, ax,e] Here, (.())() will be trained on the past records (arm contexts and rewards) Pu, from user u, and the loss function will be the
quadratic loss between the predicted reward and the actual reward. Following the Definition we will
then be able construct the exploitation graph 9 by estimating the exploitation correlation
4

---

Under review as a conference paper at ICLR 2023
based on user preferences. In (1) we consider the edge weight between nodes u, u' to be Wie (u, u') vr(1) ( (ara)), (xrL)), where (s))) :) is the mapping function mentioned in
Definition [line 11, Alg. 2).
User Exploration Network f(2), To estimate the potential gain (i.e., the uncertainty for the reward estimation) E u. Li.1
p( (xt,)), we adopt an additional user exploration network fa f(²) inspired
by Ban al. 2022b). As the confidence interval of reward estimation can be expressed
function of network gradients (Zhou et al. 2020 Qi et al. 2022). we apply s(² to directly
(-) gradient of (.) given arm Ti,t, denoted as V ffll (xr,.). For the training process, ( will
learn the confidence bound with the gradient of fu (-). Here, the input of s!. is the network
be trained with the past gradients of fil)) i.e. {V flll (xr))reTT.. as inputs; and the uncertainty of
reward prediction {r, fü (1)
(x, (a,))reT... will be the labels. Analogously, for the estimated user exploration graph G(? and given two user nodes u, u', we let their edge weight be w (u, u') (2(2) ( (V (mLL)) fo
(arta) where V Vsu (arL.) stands for the gradient of fi)
given arm E as the input (line 12, Alg. 2 and (..)) is the mapping function in Definition
Network Architecture. For the theoretical analysis and experiments, we apply separate L-layer (I. 2 2) fully-connected (FC) networks for both kinds of user networks. Their weight matrix entries for the first T. layers are drawn from the Gaussian distribution N 2/m). The entries of the last layer (L-th layer) are sampled from (O, 1/m). Complementary details are in Appendix Section C
3.2 EXPLOITATION AND EXPLORATION WITH USER GRAPHS
With two kinds of estimated user graphs encoding user correlations, we apply GNN models to separately estimate arm rewards and potential gains for a refined arm selection strategy, which enables us utilize the past records from all the users compared with single-bandit algorithms (i.e., methods with no user collaboration).
3.2.1 ARCHITECTURE GNN MODELS
In round i, with user exploitation graph 0 for each arm æ Xr apply the exploitation GNN model fann (Ii,e, G.. (ff))) ) to collaboratively estimate the arm reward fr,4 for the received user u E U. We start from learning the aggregated hidden representation over the user graph, denoted as
(1)
H agg o (((())))) (X., ))g)) R
(3)
where s/1 -D!!) (D) is the symmetrically normalized adjacency matrix of (!))
and represents the ReLU activation function. With being the network width, have O (1)
agg
Rundxm as the trainable weight matrix. After propagating the information for 1: hops over the user graph, each row of H agg corresponds to the aggregated m-dimensional hidden representation for one specific user-arm pair (u, æx,r), U. In this way, the propagation of multi-hop information can provide a global perspective over the users, since it also involves the neighborhood information of users' neighbors (Zhou et al. 2004). Here Eq. 3 the embedding matrix X₁, for the arm Ei,t X,, i [a] is defined as
x 0
0
x.
0
E Roxnd
(4)
to partition the weight matrix (1) gmn. for different users. In this way, is designed to generate individual
m-dimensional representations w.r.t each user-arm pair (u, 25,)). u E U. which correspond to the
rows of the matrix multiplication (X; (!)) gaxm.
Then, with Ho = H Hagn we feed aggregated representations to the L-layer (L 2 2) FC network as
H (H -HH111001)) (()) a Le [L L-1]. 1].
Fau(),,)) H L-1 O (1)
E R
(5)
5

---

Under review as a conference paper at ICLR 2023
where (Z.,) E R" represents the reward estimation for all the users in U, given the arm a Received user u, in round t, the reward estimation for the user-arm pair (ut, (x,e)) would be the corresponding element in P (line 8, Alg. D. represented by:
Fi. gan (ma,, L.f o = [Fau((ii)))
(6)
For the FC network, the weight matrices for the first I. - 1 layers are O E R , [1, , 1]. and for the L-th layer, we have O E RT. Here, we use O [vec( agg)", )T vec( (T... ve (Tj E IR to represent the trainable parameters of the GNN exploitation model. The exploitation GNN model fa (.) will be trained with GD based on all the received records Pr. Then we apply the quadratic loss function between the reward prediction Jgna(a, gf O (1) gmi
E
grn
} of chosen arms Ly, and the actual received rewards {r, )e(())
Connection with the Reward Function Definition (Eq. D and Constraint (Eq. 2) It is known that when width m is large enough, the FC network is naturally Lipschitz continuous with respect to the input (Allen-Zhu et al. 2019). In our case, with aggregated hidden representations H Hagy being the input to the FC network (Eq. 5 we will have the difference of reward estimations fr. bounded by the distance of rows in matrix H Hagg (i.e., aggregated hidden representations). Therefore, given arm Fi,s E X2 and two users u4, Uy E E,, the difference of their estimated rewards |[Fuu(z,,))l,,
(2,,)) can be bounded by the distance of their estimated correlation vectors (i.e, the corresponding rows in S) This matches the reward function definition and the constraint presented in Eq. E
Exploration GNN Model. To achieve adaptive exploration with user collaborations, we apply second GNN model Igin ( ((?) O .mn to evaluate the potential gain b,, of the reward estimation foin (zi,,, G O pmn) (line 8, Alg.
(2)
Here, the input is the user exploration graph G(. and the corresponding input graph signal is the gradient of the exploitation GNN model Virghakk. V Voun fann (2i,es L. (() gmm)). Analogous to fgan(( (-), the architecture of fgen (-) can
((1) also be represented by Eq. Eq. Note that while Jamn(() (-), .(2) farnn()) (-) have the same network width and o.g
number of layers, the dimensionality of weight matrices θ (1) E IR
((2) <Raxx is different.
"agg
Similarly, the exploration GNN model will be trained with GD. With the quadratic loss function, we
aim to minimize the difference between estimated potential gains <(2) [[[fgh)) of²), e (2) rE(()
and the actual ones {r, g (æ+, gf e (1) )
Instead of calculating neg-negeeive UCB intervals (upward exploration only) as in existing works (e.g.. Gentile (e.g..Gentile et al. (0014) g Ban et al. (022ag), the exploration GNN model foan (-) leverages both
gradient information from the exploitation GNN model f (-) and the user exploration correlations (i.c., ((?)) to achieve adaptive exploration (downward and upward exploration).
Remark 3.1 (Reducing Input Complexity). The input of f(2)) (-) the gradient Velgnu (x) given the arm æ, and its dimensionality is naturally = (nd x m) + (L - 1) X m² + m. which can be large when increasing the network width m and depth L.. Inspired by Convolutional Neural Networks (CNNs), e.g., RRaeeovic et al.) (2018), we apply the average pooling to calculate the approximation for the original gradient vector in practice. In this way, we can save the running time for large matrix multiplications, and reduce the space complexity at the same time. Note this approach is also compatible with user networks in Subsection 3.1 To prove its effectiveness, we will apply this
method GNB for all the experiments in Section 5
Remark 3.2 (Working with Large Systems). When facing large number of users, to deal with potentially high computational cost, we can apply approximated user neighborhoods to reduce the running time of GNB. Given user graphs (()) G!?) in terms of arm Ei,t, we derive approximated user neighborhoods (ue). (722 C 24 for target user Ut with the size (ue)) = (V:))( n, where ñ << n. For instance, can choose a subset of ñ representative users (e.g., users who always post high quality reviews in e-commerce platforms) to form A(() (())))))) A, (u,) for the downstream GNN models, which can significantly reduce the computation cost, Related experiments provided in Subsection 5.3 and Appendix Section B

---

Under review as a conference paper at ICLR 2023
ALGORITHM 1:
Graph Neural Bandits (GNB)
Input: Number of rounds T. network width m, information propagation hops k. Functions for R + R.
edge weight estimation (.;)),,)) -), (..-).
:R (-.-):
Output: Arm recommendation æt for each time step
Initialization: Initialize parameter O for all models. for 1. = 1, ..,, T do
Receive user u, and set of arm contexts X, (.La) ()) Construct two kinds of user graphs {g he((a) {9 for each ii,t E X do Compute reward estimation रi = sgnn e (Mi,e G ( g and the potential gain e gma
(2) he for arm set Nr with Algorithm
b,, f
(V
fgnn(r... (a.,, [ O gan I- G.0 O (2) len).
end 10
Play arm Zy = arg max Ex (fi. + by,r), and observe its true reward rt.-
Train the user networks f()) (())) /!²) (-: θ (22) and GNN models fgi (: gmn)
11
fann ( (2) with gradient descent, according to Algorithm gan
12
end
gna> the matrix entries of the aggregation weight matrix ② Oagg and the first L FC layers 1O..... O1-1) are drawn from the Gaussian distribution N(0, 2/ 2/m). Then, for the last layer e, we draw its entries from N 1/m).
Weight Matrices Initialization. For both GNN models (())
gan and O (2)
3.2.2 ARM SELECTION MECHANISM AND MODEL TRAINING
In round 6, with the current parameters (1) 1, for GNN models before train- ing, the selected arm chosen as De arg max ( (0 (o
(V (Vogu, form (Zi,, G [0) [ based on the estimated reward and potential gain (line 10, Alg. [.. After receiving the true reward rt, we proceed to update the user networks and GNN models based on GD and quadratic loss function (line 11, Alg. TT.. Pseudo-code summarizing the training procedure is shown in Alg, (Appendix Section D)
4 THEORETICAL ANALYSIS
In this section, we present the theoretical analysis for the proposed GNB. Here, we consider each user u E l/ to be evenly served T/n rounds up to time step T, i.e., [T... Tu.t T/n,, which is standard in closely related works (e.g.. Gentile et al. (2014); Ban & He (202T)) To ensure the neural models are able to efficiently learn the underlying reward mapping, we have the following mild assumption arm separateness.
Assumption 4.1 Sp-Serrrtss Arms), After a total of rounds. for every pair Di,t, By with t, t' E [T] and i, E [a]. if (t, i) + (t',")), we have || - 2z,e 112 2 where < S (+)).
Note that the above assumption is mild, and it has been repeatedly applied in previous works on neural bandits (Ban et al., (002b) and over-parameterized neural networks (Allen-Zhu et al. 2019). Meanwhile, Assumption 4.2 in Zhou et al. (2020) and Assumption 3.4 from Zhang et al. 202 also imply that no two arms are the same, and they measure the arm separateness in terms of the minimum eigenvalue A (with Ag 0) of the Neural Tangent Kernel (NTK) (Jacot et al. 2018 matrix, which is comparable with our Euclidean separateness p. Note that since can be manually set (c.g.. L= =),,, 2), can easily satisfy the condition 0 < (+)) as long as no arms are identical.
Based on Definition [ and Definition 3 given an arm Ci,t E Xr, we have the adjacency matri- ces !!!: and (2). for the true arm graphs g(1).x, g((2)** For the sake of analysis, given any adjacency matrix A, derive the normalized adjacency matrix S by scaling the elements of A with 1/n. We also set the neighborhood parameter = 1, and define the mapping functions (()((, (a, b), (2) b) := generalized to other mapping functions with the Lipschizz-cnntiuityy properties.
given the inputs a, b E R. Note that our results can readily
7

---

Under review as a conference paper at ICLR 2023
We proceed derive the regret bound for T time steps, denoted as R(T). Here, the following Theorem 4.2 offers the cumulative regret bound covering both types of error: (1) the estimation error of user graphs; and (2) the approximation error of neural models. Let m, J1 be the learning rate and iterations for user networks, and 7/2, J2 denote the learning rate and iterations for GNN models.
Theorem 4.2. Define E (0. 1), O Eı, S ((1/T) and 0 <
S (11LL). With the user networks defined in Eq. Zand the GNN models defined in Eq. 3 with T. FC-layers, let their width
m 2 52 Poly(T, L, a, )) 1og(1/0)).
With training process in Algorithm set parameters
P 1.
112 =
P
111 = O J1 Poly(T, n. 0, L)
Poly(T, n,
Poly(T, a,
Poly(T, a, L) p.
1
log(
J2 = =
loog
p.
Then, following Algorithm 7 Algorithm 2 for arm pulling and user group update, with probability at least 1 - 8, the T-round pseudo-regret R(T of GNB could be bounded by
log( Tn
R(T) S VT O(L) 0(L3) VT. O(12)
) +
The proof of Theorem 4.2 and the full regret bound are presented in the Appendix.
Remark 4.3 (Reducing v to (log(n), While our O(yTlogg( bound matches theoretical bound of state-of-the-art EE-Net (Ban et al. 20226), EE-Net only considers the single-bandit setting with no collaboration among users. Compared with Meta-Ban (Ban et al.] 2022a). we provide the theoretical analysis from a new perspective regarding the fine-grained user collaborative effect and GNNs. In particular, compared with existing user clustering works (e.g.. Ban et al. (2022)); /Gentlle al. 2014): [i et al. (2019) ; Ban & He (2021) imposing the additional /n (where is the number of users) factor incorporate user collaborative effects, our GNB only end up with the / Vlog(n) term by adopting GNN models for user collaboration, which is sharper than existing works.
Remark 4.4 (Removing i.i.d. Assumption). Compared with existing clustering of bandits algorithms (e.g.,.eetiile et al. (2014); Li et al. (2019); Gentile et al. (2017); Ban et al. (2022a)) and the single- bandit algorithm EE-Net (Ban et al. 20226), we avoid making the i.i.d. assumption for the arms by applying the martingale-bass basde analysis. For real-world applications, their i.i.d. assumption can be strong since the candidate arm pool is always conditioned on the received records, and candidate arms for a specific round can also come from different distributions.
5 EXPERIMENTS
In this section, we evaluate the proposed GNB framework on multiple real data sets against nine state-of-the- algorithms, including: CLUB (Gentile et al. 22114), SCLUB (Li et al. 2019). LOCB (Ban & He| (022). DynUCB (Nguyen & Lauw 2014),, COFIBA (Li al. 2016). Neural-UCB-Pol (Neural-Pool) (Zhou et 2020). Neural-UCB-Ind (Neural-Ind) (Zhou et al. 2020), EE-Net (Ban) ctt al. 2022b), and Meta-Bat Ban Ban et al. 2022a). We will include the descriptions for the baselines the Appendix Section
5.1 REAL DATA SETS
Recommendation Data Sets. "MovieLens rating dataset" https::/www. tttp://wwwwwwwww.w./t./. org/ datasets /wwvie/ovoeeeeeooooooooooooooooo eenleen//// 0m/) includes reviews from 1.6 x 10s users towards 6 x 10t movies. Since the genome-scores of user-speeified tags are provided for each movie, we select 10 tags with d = 10. Here, the user features the highest score variance to generate the movie features v, E 1R
Dn RRd, u E 24 are obtained through singular value decomposition (SVD) of the rating matrix. Here, we use K-means to divide users into 50 groups based on V., and the group information is unknown to models. In each round f, a user u, is drawn from a randomly selected group. For the arm pool X. of 10 arms, we randomly choose one bad movie (with two stars or less, out of five) rated by u, with reward 1, and randomly pick the other 9 good movies with reward O. For "Yelp" data set (https:/ www. yelp com/ dat asett, we extract ratings and build the rating matrix w.r.t. the top 2, 000 users and top 10, 000 arms with the most reviews. Then, we use SVD to extract a normalized 10-dimensional feature vector for each user and restaurant. Given the rating for specific
8

---

Under review as a conference paper at ICLR 2023
Comulative regret on Mavielants detaset
Cumulat negret on Yelp delusel
Camulaation regnet dalasel
Oumulal pye regret on
Lo
LOC3
LOCB
LOCB
$000
QUIR
4100
CJI
SC
SCJU
SCuuD
SCLU
COTTA
COTBA
COFTA
$000
0003
Mazral
Nastel
Hara
4000
Vaat
4003
Mata
403
Parts Bar
Dyn UCB
Dyr uCB
Dyr UCB
Dyr UCC
CN3
GND
GN2
CVp
2000
2000
2003
0
2000
1003
H007
1002
10007
2000
4003
6003
5003
_00000
2000
1000
19770
200000
2000
4100 10001
1030
3000
Figure 2: Cumulative regrets on the recommendation and classification data sets.
user-item pair, if the user's rating is greater than three stars (out of five stars), the reward is set to 1; otherwise, the reward is O. Similarly, we apply K-means clustering to divide users into 50 groups based on user features. In each round i, target We, is sampled from a randomly selected group. For the arm pool X4 we randomly choose one good restaurant rated by u, with reward 1 and randomly pick the other 9 bad restaurants with reward 0.
Classification Data Sets. In addition to the two recommendation data sets above, we also perform experiments two real classification data sets under the recommendation settings, which are and "Shuttle" (htt /tas:///arciiie (Shuttle) data sets. Similar previous works (Zhou et al. 2020 Ban et al. 2022a), given a sample x E Rd we transform it into C different arms, x ....., O),..., #||| = (0, 0, ...... x) E Rd+C-1 where we add C zero digits as the padding. The received reward will be 1 if we select the arm of the correct class,
"MNIST" (ht tt /ytp://yan....m www.nncccnncncm wwm wttp:///nn...o//////o/o//ea//aaa
ics. ucc. edu mm http://aaaaettttsseeeeeee
as æ1 - (z: ...... 0), 22
else the reward will be O.
5.2 EXPERIMENT RESULTS
Figure 2 iillustrates the cumulative regret results on the four data sets, our proposed GNB manages to achieve the best performance against all these strong benchmarks, First, since the MovieLens data set involves real arm features unlike the Yelp data set that includes high inherent noise, the performance of different algorithms on the MovieLens data set tends to have larger divergence. Among those regret results, the algorithms with neural architectures (Neural- PPol,, EE-Net, Meta-Ban) generally perform better than linear algorithms due to the approximation power of neural networks. However, as Neural-Ind considers no collaboration among users, performs the worst among all baselines on these two data sets. EE-Net outperforms Neural-Pool thanks to its adaptive exploration strategy. For classification data sets, Meta- performs better than the other basclines by modeling user correlations under the non-linear setting. Different from recommendation data sets, the classification data sets involve more complicated reward mapping functions, and this might lead to the poor performances of linear algorithms. Our proposed GNB consistently outperforms all baselines by modeling fine-grained user correlations and utilizing the adaptive exploration strategy simultaneously. In addition, GNB only takes most 75% of Meta-Ban's 's running time to finish the experiments (Table D. since Meta-Ban trains the framework individually for each arm before making predictions.
5.3 SUPPLEMENTARY EXPERIMENTS
Due to the page limit, we present additional supplementary experiments in the Appendix Section B including: (1) experiments on additional data sets: (2) with increasing number of users, experiments demonstrating the effectiveness of applying approximated user neighborhoods (Remark 3.2 (3)
experiments showing the potential performance impact on GNB when there exist underlying user clusters; (4) the parameter sensitivity study showing that our adaptive exploration strategy can indeed improve the performance of GNB, and the effects different hops & for information propagation.
CONCLUSION
In this paper, propose novel framework named GNB to model the fine-grained user collaborative effects. Instead of modeling user correlations through the estimation rigid user groups, we estimate the user graphs to preserve the pair-wise user correlations for exploitation and exploration separately, and utilize individual GNN-based models to achieve the adaptive exploration. Moreover, under standard assumptions,
also demonstrate the improvement of regret bounds over existing methods from a new perspective of "fine- graeeerrrdd"' user collaborative effects and GNNs. Extensive experiments conducted to show the effectiveness of our proposed framework against strong baselines.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yasin Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances neural information processing systems, 24:2312 1222220, 2011.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. International Conference on Machine Learning, pp. 242-252. PMLR, 2019.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256 2002.
Yikun Ban and Jingrui He. Local clustering in contextual multi-armed bandits. In Proceedings of the Web Conference 2021, pp. 2335-2346, 2021.
Yikun Ban, Jingrui He, and Curtiss B Cook. Multi-facet contextual bandits: A neural network perspective. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 35-45, 2021.
Yikun Ban, Yunzhe Qi, Tianxin Wei, and Jingrui He. Neural collaborative filtering bandits via meta learning. arXiv preprint arXiv:2201. 13395, 2022a.
Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. Ec-net: Exploitation-exploraiion neural networks in contextual bandits. In International Conference on Learning Representations, 2022b.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in Neural Information Processing Systems, 32: 10836, 2019.
Nicolo Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory, 50(9):2000-2057, 2004.
Nicolo Cesa- Cesa-Bianchi, Claudio Gentile, and Giovanni Zappella. A gang of bandits. In NeurIPS, pp. 737-745, 2013.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling. arXiv preprint u 10247, 2018.
Wei Chu, Lihong Li, Lev Reyzin. and Robert Schapire. Contextual bandits with linear payoff functions. In AISTATS, pp. 204, 2011.
Aniket Anand Deshmukh, Urun Dogan, and Clay Scott. Multi-task learning for contextual bandits. In NeurlPS, pp. 4848 4856, 2017.
Audrey Durand, Charis Achilleos, Demetris lacovides, Katerina Strati, Georgios D Mitsis, and Joelle Pineau. Contextual bandits for adapting treatment in mouse model of de novo carcinogenesis. In Machine learning for healthcare conference, pp. 67-82. PMLR, 2018.
Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations, 2019.
Claudio Gentile, Shuai Li, and Giovanni Zappella. Online clustering of bandits. In ICML, pp. 757-765, 2014.
Claudio Gentile, Shuai Li, Purushottam Kar, Alexandros Karatzoglou, Giovanni Zappella, and Evans Etrue. On context-dependent clustering of bandits. In ICML, pp. 1253-1262 2017.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In WWW. pp. 173-182, 2017.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 639-648, 2020.
10

---

Under review as a conference paper at ICLR 2023
Joey Hong, Branislav Kveton, Manzil Zaheer, Yinlam Chow, Amr Ahmed, and Craig Boutilier. Latent
bandits revisited. Advances in Neural Information Processing Systems, 33:13423- 13343,, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Parnian Kassraie, Andreas Krause, and Ilija Bogunovic. Graph neural network bandits. arXiv preprint arXiv:2207.06456, 2022.
Lihong Li, Wei Chu, John Langford, and Robert Schapire. A contextual-bandi approach to personalized news article recommendation. In www, pp. 661-670, 2010.
Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. Collaborative filtering bandits. SIGIR, pp- 539-548, 2016.
Shuai Li, Wei Chen, Shuai Li, and Kwong-Sak Leung. Improved algorithm on online clustering of bandits. In IJCAI, pp. 2923-2929, 2019.
Odalric-Ambrym Maillard and Shie Mannor. Latent bandits. In International Conference on Machine Learning, pp. 344.44. PMLR, 2014.
Trong T Nguyen and Hady W Lauw. Dynamic clustering of contextual multi aulti--rmmed bandits. In Pro- ceedings of the 23rd ACM International Conference Conference Information and Knowledge Management, pp. 1959-1962, 2014.
Yunzhe Qi, Yikun Ban, and Jingrui He. Neural bandit with arm group graph. arXiv preprint arXiv:2206.03644, 2022.
Filip Radenović, Giorgos Tolias, and Ondřej Chum. Fine-tuning emn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence, 41(7):1655-166 2018.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018.
Sohini Upadhyay, Mikhail Yurochkin, Mayank Agarwal, Yasaman Khazaeni, and Djallel Bouneffouf. Graph convolutional network upper confident bound. 2020.
Michal Valko, Nathan Korda, Rémi Munos, Ilias Flaounas, and Nello Cristianini. Finite-time analysis of kernelised contextual bandits. In Uncertainty Artificial Intelligence, 2013.
Soffa S Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges. Statistical science: a review journal of the Institute of Mathematical Statistics, 30(2):199, 2015.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. In Proceedings of the 42nd international ACM SIGIR conference Research and development in Information Retrieval, pp. 165-174, 2019.
Max Welling and Thomas N Kipf. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference machine learning, pp. 6861-6871. PMLR, 2019.
Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in collaborative environment. SIGIR, pp. 529-538, 2016.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks, In International Conference Machine Learning, pp. 5453-5462. PMLR, 2018.
11

---

Under review as a conference paper at ICLR 2023
Keyulu Xu, Mozhi Zhang, Stefanie Jegelka, and Kenji Kawaguchi. Optimization of graph neural networks: Implicit acceleration by skip connections and more depth. In International Conference on Machine Learning. pp. 11592-11604 PMLR, 2021.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pp. 974-983, 2018.
Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In International Conference on Machine Learning, pp. 7134-7143. PMLR, 2019.
Weitong Zhang, Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural thompson sampling. In International Conference Learning Representations, 2021.
Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NeurIPS, pp. 321-328, 2004.
Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning, pp. 11492-1 11502. PMLR, 2020.
12

---

Under review as a conference paper at ICLR 2023
A RELATED WORKS
In this section, we briefly review the existing works related to our proposed GNB framework. Assuming the reward mapping be linear, linear upper confidence bound (UCB) algorithms (Chu
eet al., [ Li et al. 2010, Auer et al. 2002] Abbasi-Yadkori et al.) 2011) were first proposed to solve the exploitation-exploaateee
dilemma. After kernel-baset methods (Valko et al. 2013
Deshmukh et al. 2017 were used to address the non-linear setting where the reward mapping is the kernel-based function, neural algorithms (Zhou et al., 2020 Zhang et al. 202T Ban et al. 2021 have been proposed to utilize neural networks to estimate the reward function and confidence bound.
Meanwhile, AGG-UCB (Q al. 2022) adopts GNN to model the arm group correlations. GCN-UCB (upadhyay et al. 2020) manages to apply the GNN model to embed arm contexts for downstream linear regression, and GNN-PE (Kassraie al. 2022) utilizes the UCB based on information gains to achieve exploration for classification tasks graphs. Note that the above neural algorithms with UCB-based exploration strategy all suffer from the space complexity Ovr²) ) to store their gigantic gradient matrix, where p is the number of model parameters. This space cost is especially enormous when you increase the quantity of model parameters by adding more network width m and depth L. Instead of UCB. EE-Net (Ban et al. 2022b) achieves adaptive exploration by using neural models for estimating prediction uncertainty. Assuming a finite number of arms, (Maillard & Mannor 2014 Hong et al. 2020) discuss the latent bandits where there exist latent states that affect the reward generation. Nonetheless, all of these works fail to consider the collaboration effects among users under the real-world application scenarios.
In order to model the user correlations, (Wu et al. 2016; Cesa-Bianchi et al. 2013) assume the user social graph is known, and apply an ensemble of linear estimators. Without the prior knowledge of user correlations, CLUB (Gentile et al. (0144) introduces the user clustering problem in contextual bandits with the graph connected components, SCLUB (Li et al. 2019) adopts dynamic user sets and applies set operations to update user clusters, and DynUCB (N Nguyen & Lauw 2014) assigns users to their nearest estimated clusters. Then, CAB (Gentile al. 2017) studies the arm-specific user clustering, and LOCB (Ban & He| (2021) estimates soft-margin user groups through a random-seed based approach. COFIBA (Li et al. 2016) utilizes user and arm clustering for collaborative filtering. Apart from these linear algorithms,
note a concurrent work Meta-Ban (Ban et al.. (022a) which applics a neural meta- meta-model to adapt to different user groups. However, all algorithms mentioned in this paragraph consider rigid user groups, where users from the same group are treated equally with no internal differentiation.
GNNs (Welling& & Kipi 2017 Chen et al. 2018 Wu et al. 2020 Satorras & Estrach 2018) are a kind of neural models operating on the graph data, and have been proved effective for various tasks, e.g., community detection (You al. 2019) and recommender systems (Ying et al. 2018). In this work, we leverage GNNs to learn from user correlations and arm contexts simultaneously.
2019 Gasteiger et al. 2019 He et al.
B EXPERIMENT SETTINGS AND SUPPLEMENTARY EXPERIMENTS
B.1 BASELINES AND EXPERIMENT SETTINGS
The descriptions for our nine baseline methods are:
CLUB (Gentile al., 2014) regards connected components user groups out of the
estimated user graph, and adopts a UCB-type exploration strategy;
SCLUB al. 2019) estimates dynamic user sets as user groups, and allows set operations for group updates;
LOCB (Ban & He 202 applies soft-clustering among users with random seeds and choose the best user group for reward and confidence bound estimations;
DynUCB (Nguyen& & Lauw 2014) dynamically assigns users to its nearest estimated cluster.
COFIBA [i et al. 2016) estimates user clustering and arm clustering simultaneously, and ensembles linear estimators for reward and confidence bound estimations;
Neural-Pool adopts one single Neural-UCBB (Zhou et al. 2020 model for all the users with UCB-type exploration strategy;
13

---

Under review as a conference paper at ICLR 2023
Neural-Ind assigns each user with their separate Neural-UCB (Zhou al. 2020 model;
EE-Net (Ban et al. 2022b) achieves adaptive exploration by applying additional neural models for the exploration and decision making;
2022a) utilizes individual neural models for each user's behavior, and applies meta-moder to adapt to estimated user groups.
Meta-Ban [Ban et al.
Baseline Settings. For all the UCB-based baselines, choose theirs exploration parameter with grid search the range {0.01, 0.1, 1} individually. And we set the
for all the deep learning models, 100. The learning rate of all neural algorithms are selected by grid search in range {0.0001, 0.001, 0.01). For EE-Net, we follow the default setting in their paper by using hybrid decision maker, where the estimation is fi we apply an additional neural network for decision making afterwards. For Meta- Ban, we follow the settings their paper by turning the clustering parameter y through grid search {0.1, 0.2, 0.3, 0.4]. For GNB, we choose the k-hop user neighborhood & E {1, 2, 3} with grid search. Reported results are the average of 5 runs.
and set the network width
for the first 500 time steps, and then
B.2 EXPERIMENTS ON ADDITIONAL DATA SETS
Due to the page limit in the main body and to better compare our GNB with the bench- marks, here, we include the experiments on two additional classification data sets in this subsec- tion. They are: (1) the "Letter" data set with C 26 different classes (https: /ttp:///rcciiie ics.uci htt..edddmmmddeee http:///t/eeeettttte//eeeeeeeeee
and (2) the "Pendigits" data set uci edu http://daaaaaaaass/////////////s /ttpl//aaasssseeeeaasseees P
with C = 10 classes (httpss://archivee.crr
under the recommendation settings. Analogous to settings of the "MNIST" and the "Shuttle" data set, consider each class to be user. Given sample x E Rdd we transform into arms for different classes similar to previous works (Zhou et al.
2020
Ban et al. 2022a), namely æ1 (x:, ..... 0), 22 (0, x. ..... 0).-.. ......... .C (0, 0, ... .................... x) where additional C zero digits are added as the padding. The reward will be = if we
R
choose the correct arm that represents the sample's true class; otherwise, the reward will be 0.
Cumulative regret Letter
regret on Pendigits dataset.
7000
CLUB
SCLUB
6000
SCLLB
COFIBA
COFIBA
6000
Neurai-Fool
5000
Neura-PPol
Neural-ard
8000
Dyn.UCE
Meta-Ban
Dyn UCB
Meta
GNB
GNB
1000
2000
6000
8000
10000
2000
4000
6000
8000
1000
Figure 3: Cumulative regrets on the two additional classification data sets.
The experiment results for these two additional data sets are presented in Figure It is worthwhile to note that EE-Net continues to outperform the two Neural-UCH baselines, which is also another evidence of the effectiveness of the adaptive exploration strategy. On the other hand, our exploration strategy inspired by EE-Net further incorporates the user exploration graphs to exploit the encoded "fineggraine"" user collaborative effects. Therefore, analogous to the experiment results in the main body (Figure (2), our proposed GNB framework consistently outperforms the other benchmarks by exploiting and adaptively exploring the "fine-grained" correlations among different classes at the same time.
14

---

Under review as a conference paper at ICLR 2023
B.3 EFFECTS THE ADAPTIVE EXPLORATION AND EFFECTS OF INFORMATION PROPAGATION HOPS
In order demonstrate the necessity of the adaptive exploration strategy, we consider alternative arm selection approach (different from line 10, Alg. T
at each time step 4, with the following form:
(FED) (vi,,, G O ganle-1)
at arg max aii
(1) fgl (ai,e G (0 G(2,)
: gemle-1)
gnn
given the candidate arm set X - ( and the model parameters [O b- 10 g0gn Here, we introduce an additional parameter a [0, as the exploration coefficient control the levels of exploration (larger the a values will lead to higher levels of exploration). And we will show the experiment results with (0, 00, 0.3, 7 1.0) on the "MNIST" and the "Yelp" data sets.
Cumulative rogret Yelp dataset
Cumulative regret on MNIST dataset
0-0
a=0
0.1
6000
1=1
d=1.0
Oum reg
$ac0
8000
2000
500
0
Time stop
Time
Figure 4: Cumulative regrets for different exploration coefficients a.
In Figure we illustrate the effects of different exploration coefficients. Regarding the results in the left figure ("Yelp" data set), the adaptive exploration indeed helps to improve the performance GNB, but the performances of GNB do not differ dramatically with different values. As the "Yelp" data set contains inherent noise, the curve of cumulative regrets (including cumulative regrets of other benchmark algorithms) tends follow near-linear growing rate. However, our carefully designed adaptive exploration strategy based on user exploration graphs is still helpful to improve the overall performance, and this is validated by the fact that setting a = will lead to better performance compared with the situation when no exploration strategy is involved (a 0). On the other hand, based on the figure on the right hand side ("MNIST" data set), different a values tend to have relatively divergent results. The reason can be that in the "MNIST" data set, the mapping from arm contexts to the rewards is more complicated compared with that of the "Yelp" data set. Thus, the adaptive exploration strategy able to prominently improve the performance of GNB by flexibly estimating potential gains of different classes with the estimated "fine teine-graindd" user (class) correlations.
Recall that there exists parameter k for the GNB framework in Eq. which controls the user neighborhood hops that the two GNN models learn from. In this subsection, we will present the experiment results with & E 2,,2,3) 3} on the "MNIST" data set and the "Yelp" data set, which are presented in F
Based on the results on the two data sets, we can observe that setting k = 1, namely making the GNB learn directly from the 1-hop neighborhood, tends to yield the best result. This might be due to the fact that since our user graphs are staying as connected graphs while the user correlations are encoded by the edge weights, learning directly from the neighbor would good enough. And the pair-wise user correlations between the target user and every other user have already been taken into consideration. Meantime, with larger values (1: = 2)3), raising the matrix to the power of even entry values across the adjacency matrix, which over-smon over-smoothing problem (Xu et al., 2018 202T). The figure on the right hand side ("MNIST" data set) may support this claim. Since has already been shown in the Figure 2 that applying single
k would lead to
be related to the
15

---

Under review as a conference paper at ICLR 2023
Cumulative regret Yelp dataset
Cumulative regret on MNIST dataset
X-1
2500
7000
t=2
km2
6000
O reg
3000
1000
2000
0
Time stop
Time
Figure 5: Cumulative regrets for different neighborhood hops k.
estimator across all users (classes), i.e., Neural-Pool and EE-Net, will lead to poor performances, the "MNIST" data set tend to have complex correlations among different classes. In this case, when we increase k, different user pairs tend to have similar correlations because entries of the adjacency matrix become more close each other, which may lead to extra estimation error.
MNIST dataset
MNIST dataset (k=2)
MNIS dataset (k=
0-0.0
2500
2-0.0
0.0
2000
0:3.3
2
Cumeeetvee regret
o-1.0
2000
2000
1000
1000
1000
500
500
500
2000
4000
2000
6000
10300
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
stop
Figure 6: Cumulative regrets for different neighhorhood hops & and exploration parameter a for the MNIST data
Moreover, also conduct the experiments the MNIST data set with different sets of and parameters jointly, as shown in Figure 6 Following our conclusion above, setting A: generally leads to better results and the adaptive exploration strategy offers considerable help to improve the GNB's performance. One phenomenon to note that when we increase the value of parameter k. the performance difference of GNB with different a values will shrink. One reason for this situation is that when we increase the & value, the propagated adjacency matrix of the user graph will become more "smooth", which makes the users closer each other in terms of similarity. In this case, the effect of the adaptive exploration strategy can be affected as the user correlations estimated are less divergent.
B.4 EXPERIMENTS WITH DIFFERENT NUMBER OF UNDERLYING USER GROUPS
MovieLers dataset (5 users groups)
MovieLers data set (10 users groups!
Moviel ens dataset (15 users groups)
7000
LOCB
Dynüce
7000
DynUCB
LOCB
7000
8000
Dynuce
LOCB
6000
CLUB
6000
CLUB
5000
5000
Ncural
6000
GNB
4000
GNB
GNB
3000
1000
1000
5000
6020
6000
Time
Figure 7: Cumulative regrets for different number of underlying user groups (MovieLens data set).
16

---

Under review as a conference paper at ICLR 2023
Yelp data set (5 users
Yelp dataset (10 users groups)
Yelp dataset (15 users groups)
80000
LOCB
PynucB
8000
LOCB
DynUCa
8000
Rynucn
LOCB
Cumueetie 1e3
6000
Meta-Ban
6000
NeuraFUCB-HH
Neural-UCB-Pod
Neural-UBB-Pod
ZNB
GMB
FNN
4000
4000
2000
4000
2000
6000
10300
2000
40000
6.000
2000
10000
2000
4000
00000
81000
10000
step
Figure 8: Cumulative regrets for different number of underlying user groups (Yelp data set).
To better understand the influence of potential underlying user clusters, we conduct the experiments on the MovieLens and the Yelp data sets, with controlled number of underlying user groups. The underlying user groups are derived by using hierarchical clustering the user features, and maintain approximately a total of 50 users. Here, we apply four representative baselines with relatively good performances, which are DynUCB (Nguyen & Lauw (201)) [fixed number of user clusters].
LOCB (Ban & He 2021) [fixed number of user clusters], CLUB (Gentile et al.] (0144) ciistance-basee user clustering). Neural-UCB-
(Zhou al. 2020) [neural single-bandit algorithm). and Meta-Ban (Ban et al. 2022a) [neural user clustering bandits]. In particular, DynUCB and LOCB are provided with the true eluster number the prior knowledge to determine the quantity initial user clusters / random seeds. The experiment results are shown in Fig. 7 and Fig. g
from the results, our proposed GNB consistently outperforms other baselines across different data sets and number of user groups. In particular, with more underlying user groups, the performance improvement of GNB over the baselines will slightly increase, due to the increasingly complicated user correlations. The modeling of fine-graines user correlations and the representation power of our WWNN-based architecture can help explain GNB's s good performance, and the ability of utilizing user correlations.
As
B.5 EXPERIMENTS WITH APPROXIMATED USER NEIGHBORHOOD
In this subsection, we conduct experiments to support our claim that applying approximated user neighborhoods is feasible solution for increasing number of users (Remark (.2). Then, we consider three scenarios where the number of users (200, 300, 500J. Meanwhile, we let the size of the approximated user neighborhood. AT() ( fix to ñ (w)) [A)())( 50 for all these three experiment settings, and the neighborhood users are sampled from the user pool 24 for experiments.
MovieLens dataset (200 users)
MovieLens dataset (300 users)
MovieLens dataset (500 users)
8000
Neural-Pool
8000
Neural-Pool
8000
Neurai-Pool
Dynlcc
DysUCh
Bynüch
LOCB
6000
LOCB
6000
LDCB
Comuetive
SCLUB
Mata-Ban
Muta Ban
Mata-Ban
2000
ERI
Neuralard
2000
Ghab
Neural-Ind
2000
CNB
Neuraland
0
Time stop
Time step
Time stop
Figure 9: Cumulative regrets for different number users with approximated user neighborhood (MovieLens data set).
The experiment results are shown in Figure 9 Here, the baselines with increasing number of users. In particular, given a total of 500 users, the approx- imated neighborhood is only 1/10 (50 users) of the overall user pool. These results serve as a clear support that applying approximated user neighborhoods (Remark) 3.2p is a practical way to scale-up GNB in real-world application scenarios.
that the proposed GNB still outperforms
17

---

Under review as a conference paper at ICLR 2023
B.6
RUNNING TIME ON REAL DATA SETS
Data sets (10* rounds)
Methods
MovieLens
Yelp
MNIST
Shuttle
CLUB
230
359
SCLUB
274
82
363
4
LOCB
215
31
223
3
DynUCB
214
29
357
3
Neural-Pool
509
321
289
226
Neural-Ind
472
281
265
169
EE-Net
2435
2149
2903
2052
COFIBA
321
135
11874
13
Meta-Ban
20170
19825
18172
18101
GNB (Ours)
14121 [4295]
12506
[4082]
4299 [1072]
1606 [185]
Table Average running time results (seconds) on real data sets. The running time the brackets
"[J" is the actual time consumption for recommendation w/o the time consumption for training.
From Table 1 we see that compared with the most closely related work, Meta-Ban, proposed GNB is generally faster, since GNB does not required to re- re-train the model for each candidate arm.
Although the other baselines, especially the linear baselines tend to run much faster compared with our proposed GNB, their experiment performances (Section (5) are also not comparable with our proposed GNB their linear assumption is too strong for most application scenarios. In particular, for the data set with large arm context dimension d, the mapping from the arm context to the reward will be much more complicated. In this case, shown by the experiments on the MNIST data set (d = 784) in Figure 2 the neural algorithms manage to achieve an undoubtedly huge improvement over the linear algorithms, and have the reasonable running time.
Here, the numbers in the brackets "[]" are the time consumption for the actual recommendation process. We have the following remarks: (1) Based on the running time in the brackets, we see that for the two recommendation tasks, GNB takes approximately ~ 0.4 second per round to make the arm recommendation for the received user, which is reasonable in real-world cases; (2) In all the experiments, we train the GNB framework per 100 rounds after T >> 1000 and still manage to achieve good performance. Thus, the running time of GNB in long run could be further significantly improved by reducing the training frequency since have already have enough data and accurate framework; (3) Moreover, since we are actually predicting the rewards and potential gain for all the nodes within the user graph (or the "approximated" user graph), GNB is able handle multiple users in each round simultaneously without running the recommendation procedure multiple times, which efficient in real-world cases.
18

---

Under review as a conference paper at ICLR 2023
C USER NETWORKS ARCHITECTURE.
Here, we can choose different architectures for fu ()), (2) to deal with various application scenarios (e.g., Convolutional Neural Networks [CNNs] for recommendation tasks of visual contents). In this paper, for the theoretical analysis and experiments, we apply separate L-layer fully-connected (FC) networks for user exploitation models and exploration models, as
fu(:: 0,0(0).-10@LL-2... ((1x))))
with θ, 1v) ..... vec [ being the trainable parameters, and being the ReLU activation. Here, since f (), (-) either the arm a or the network gradient V V full
(.) are both L-layer networks shown Eq the input X can be
(.))
Initialization. Then, the weight matrix of the input layer is different for two user networks where o Rxd and ( E Rmxp, The rest of the layers will the same comparing the two kinds of user networks, which are O E Rmmxm E 1., L - 1]. and θ E IR 1x
D PSEUDO-CODE FOR ESTIMATING USER GRAPHS AND TRAINING THE GNB FRAMEWORK
ALGORITHM 2: Estimating Arm-Specific User Graphs
Input: Model parameters O Functions for edge weight estimation
(,,), (3) R x R + R.
Output: Updated user graphs (9 beld (0(2) hela).
Initialize [ [e(]]. ( heja].
for each user u E ll do for each arm Ti,t E Xr i E [a] do
Compute Pu,i = (1] (a [0))) 1-1), and
bu, 12))( (1) (1) ( ( 1-1); ( 1-1)
end
end
for each arm Di, E Ne do
for each user pair (u, u() E 1 x 11 do
For edge weight (1) (u, w/) E W
update (1 (u, w²) = (((1 (Pusss
11
(2) For edge weight w (u, u') E ( update w u') = (2) (B..i. b). end
(2)
12
13
14 end
15 Return user graphs ( Jaelaj ( helaj-
19

---

Under review as a conference paper at ICLR 2023
ALGORITHM 3: Model Training
Input: Initial parameter Op step size 7,, 12. training steps J1, J2, network width m. Updated
user graphs 9("), g.a²) Served user Up.
Output: Updated model parameters [6
(1) oz
O ue (., [ (1) and [e (0(2) ganle-
(2)
[ Jt. [ (2)
User-Modell-Tainingg (u+, [ (1)] Jo, [ Jo).
for Vu' E U, u' + ue do
0 (1)
O h--1
e (2)
1-1
eend
[0(0]
ganle 9 2)
g GNN-Model GNN-Modde-Trainnng (lo (19g)n)
| (2)
gnnlo).
Return
[ (0_) O (2)
)r.
gmm.t.
(2)
Procedure User-Model- Training (u,, o (1)
(2) (o)
[0
e 0.
+ (
o (2) 0-
# Training of fu (-) (1))
11
Let [(O!.) = ΣrET,
c,; (.)) - rr rr1
12
for j = 1,2,
do
1 (0())p--1 171 V
14
end
# Training of ((2)(
16
Let L(O (2) =
17
ExeT,,
V
@ Su (11)
(=: - (r+ f(()) (xri 10 [+-))-
(xri ( I1:; 0
18
for j = 1,2, .......JJ. do
2 [
(2)
1/1 VoL((
(2(2))))))
19
Me
29
end
Let - o 11
- o (2)
Sample and return new parameters (00 (1)
(3 (1) (166)) Ir. ( re((-
22
23
end
Procedure GNW-Model-Training (lO (1)
gni n. + (2)
gmn
25
(0 1o
gan]"
gnn
(1)
go.
gnn. gnn.o.
(2)
Training of f (.)
26
Let L(O grm Σrejel
(1)
(1))) ( ( (
gnn)
28
for jo 1,
Jydo
(0) 1 = (1)
1--1 122 VeL( 1
29
gnn.
grn.
30
end
Training of f (.)
31
Apply pfl (x,) denote (1)) rr gf (1), (1) (mm]--1) 1).
32
Let [(e mmn ΣtE Jann
(2)
(1) sonn (I+)), _.
θ (2) - (rr forn 0(,1))
gnn)
34
for j = 1,2, J2 do
O (2)
(0(2)
122 V ( gmml
(2) 1-1) 1-1)
35
end
37
Let g
o gnn.
(1) 12
g f O (2) J/
gran
Sample and return parameters (0
gon L, O gemmle) ((16()) (1. @ g
(1)
1- rE((]-
end
20

---

Under review as a conference paper at ICLR 2023
E PROOF OF THEOREM 4.2
Before presenting the regret bound after T rounds, we proceed to bound the regret at a single time step 1 E [T]. Recall that there are two kinds of user graphs (g hell) t Jiejal at each time
step 4, while we can also build true user exploitation graph {g (1) Re(u and true user exploration graph (C["""" Heew based on the Definition | and Definition respectively. Comparably, the true normalized adjacency matrices of G
. " [a] are represented S!
With rt, rf separately being rewards for actual selected arm 2; E 2r and the optimal arm o E Xz we formulate the pseudo-regret for a single round as R2 Eir [ug X,] -K(re lu,, Xe] based on the candidate arms X₂ and received user 4e for the current round t. Here, regarding our arm pulling mechanism in Algorithm 1 gnn (VS, (3rx), G.2); 1 given the selected arm a; with the input gradient V (ax))
we have (onn((;))
ggnn (0 - (m)l--1)
fine (me, (" cg
(cg as the normalization factor, such that V (1) (a:) l| - 1), and
the estimated user graphs g g related to x,, Analogously, we also have estimated user graphs g 0 for the optimal arm x;.
Then, in round t [T], the single-round regret Re can be bounded as
R₂ E[r]] lu, Xz] B[re|ue, Xc]
E[r] [u, Hc] - (gnn((:)) + synn (2x) - Uer Xz]
Err ( Xi] fann(zt) + (onn((x)) - BBro| ue, Xz]
S El [ine- - fgnn (z;) Uu Xe
+ E |Ire - fgnn uue Re,
E | gnn! (V.e (1) (arz), G
(2) g 1) - (ri - - (2i G1!1 ( ( (-1) ut, Xe +
E [ (V (1) (x.); g (2) l--) - (re - -Jggh (II, g. (1)
gun le -1-) Ut, Xe
= =B.(xt)) + (x,)
where inequality (i) is due to the arm pulling mechanism, i.e., fynn (Ix) 2 sgnn (x;). and (B+(,) the regret bound function round & formulated by the last equation. Then, given an arbitrary candidate arm X, with reward r, and its estimated user graphs ())) we have
(mr
E.
(z), g(2); l--))- - -(r -r-- (WWn (ar, (()); 1 ( [لا
Me, 2g
E
|
(z), (2)), (2) gaan ( 1--1) (r faln (z, g()); e (1) ((mlle=)))
we 2e
E
L (2,, g(0)+ [ Sgin (z, (()); 10 (0 gmn Ir- )) Me. Xi +
E
(22) (mx), g(2)* O p 122h (VJ{ (1), (z:), g( ( (2)
k-)
1)
ue, Xe
+
1 (V)( (())) (æ), g(2); [ (2) (2mlk-1) - ((2) V (x),
((
10 (2) (- Ues Xe
8
Here, we have the term / representing the estimation error induced by the GNN model parameters ( (1) 11, (2) 1 1), the term /2 denoting the error caused by the estimation of user exploitation
21

---

Under review as a conference paper at ICLR 2023
graph. Then, enror term Ig is caused by the estimation of user exploitation graph, and term I4 is the output difference given input gradients (((")) (a:) and ( (zz), which are individually generated
by true user exploitation graph g()) and the estimated exploitation graph g()).
These four terms /: 12. 13, 14 are respectively bounded by Lemma G.2 (Corollary G.3 and the
bounds in Subsection G.T) Lemma Lemma (.3 and Lemma G.7 in the appendix. Then, with the notation from Theorem 4.2 the pseudo regret after T rounds, namely R(T), can be bounded by
R(()
Σ Rs BET2]
31.
Tn.a
2. Vi. (V +
+ (1 + (1+)2) 2 2
V2
8
+ +(1+01 tL log 5/6 (m)
(m))
L log(m)) 0
+ L log (1/0((m))) (m)
0
1/26
pvm
pa/3m22/ 2/6
3L
Tn
+2. O(L) V81. (v25) +
+ (1 + (1+7))1
v) 2 2log(
V2
+ +0 1L9 1.LL'og' log
2/
(m) (m)
3L
Tn Tn-a)) + 4r,
+ O(L') Vst (V2g1 +
+ +1+ + 21 2
p1/3m22/6
v2
8
3L
+ (1+ + 722) logg Tn.a
R(T) 22 VTV 25 + (12
v2
3L
+ (1 + m) 2log( Tn + +01)
+ .VT.0(13) (V2g +
V2
$ VT. ((VE5+0(L2)) 2 +
Tn. a 8
2 2log(
+ ((L)+ (1
where the second inequality is because we have sufficient large network width m >
@ Poly(T, I,, a, ()- 1og(1/5) as indicated in Theorem 4.2 Here, since m 2 (((Poly(T)) terms 7, can also be bounded by (()). Therefore,
R(T) S VT. ( (VV55+ + +0(2)+++(()) 2 ( 2) VT O(L)+
$ VT. (0(12)) 2 ()00)))))))))))0))))))))))) + +0.)+++001)))) 0(1) 8
= =VT -(T- + VT O(13) log( - +
Tn .n.a )
when we have &2 - ( The proof is then completed.
Apart from the two remarks in the main body (Remark 4.4 4.3
we also want to mention another below.
improvement over existing works with the Remark E.I
Remark E.1 (Removing d, d Terms). Existing neural single-bandit (i.e., with no user collaboration) algorithms (Zhou al. 2020; Zhang al. 2021) derive the bound (dd/T log(T)) based on neural gradient mappings and ridge regression, and they involve the effective dimension term d of the NTK matrix, which can grow along with the scale of network parameters and number of rounds T. The
linear user clustering algorithms (c.g., Li et al. (2019): ; Ban & He (022I): Gentile et al. (0177) have the bound Odv' log(T) with the term arm dimension d, which can be large given arm contexts in the high-d nigh-dimensional space. Here, of / /To and remove the dimension terms d, d. We apply the generalization bound for over- parameterized neural networks (Allen-Zhr et al. 2019 Cao & Gu 2019) instead of regression-basse analysis to remove the /log(T) term, and the generalization error is also unrelated to or d for over-parameterized neural networks.
improve their bounds by multiplicative factor
22

---

Under review as a conference paper at ICLR 2023
F GENERALIZATION OF USER NETWORKS AFTER GD
(P (:0 a e U. Up to certain time step and for given user E U, we have all its past arm-reward pairs Pu,r-1 = {(x,, Ty Te)TTT.. Before presenting the bounds, with two vectors ã; a as the input such that | 1, 2 1, inspired by (Allen-Zhu et al. 2019). first define the the following operator
In this section, we present the generalization results of user networks
( x)
V2 2 cc
as the concatenation of the two vectors v2 2
æ and one constant c, where c V
And this operator makes the transformed vector llő(ā, æ)1l2 1. The idea of this operator is to make the gradients V
"e!" e(!) (:3 ((!)) of the user exploitation model, which is the input of the
exploration model f! (.), comply with the normalization requirement and the separateness assumption (Assumption 4.1). For the sake of analysis, we will adopt this operation the following proof. Note that this operator is just one possible solution, and our results could be easily generalized to other forms of input gradients under the unit-length and separateness assumption. Similar ideas are also applied in previous works (Ban al. 20226).
F.1 USER EXPLOITATION MODEL
With the convergence result presented in Lemma F.6 we could bound the output of the user exploitation model faa)) (.) after GD with the following lemma.
Lemma F.1. For the constants E (0, (1,)) and &1 E (0, given user u E U and its past records Pu,-- up time step t, suppose m, JJ satisfy the conditions in Theorem 4.2 Then, with probability at least 8, given an arm-reward pair rx, we have
(x;
10)1 Sm
11)
where
L
L244
?1 = 2+0
log m
+0
n² p4p4/3m1/6 log log!1/6 (m0)
n mm
(1)
(1)
Proof. For brevity, we use
to denote
The LHS of the inequality could be written as
Is (x: 1 (1)
$| (x:;
( (x: +
(1)
6 (1),
1o),
@
_ Jo>i
| (2)
+ + (
(o) +
(o;
Jo).
_e.") 1o)l-
Here, we could bound the first term on the RHS with Lemma F. second term, and recalling _ - [ Jolla S w, would give
Applying Lemma F.8 on the
1 [ 5 2 V 6.) (x: Q (1), (1) -
@
10
O 1 m log(m)) | @0 Jollz
22+0 1 [
1o 1o + O(L2 [)) 11 [0"
Then, with Tu, 1. applying the conclusion of Lemma F.6 would lead to
(Tint) pvm logm)
( (Tunt) 3 log m) PVm.
10 (z: (x; 6 22+0(++
- V llgmm))
[3L log m
L
=20
n² p.m
+ +
n² p ml 1/6
log 11/6 1mmmm)
= 7.
Then, under the assumption of arm separateness (Assumption 4.1. estimation error of the user exploitation network (( (:: (1) in the current round f.
we proceed to bound the reward
23

---

Under review as a conference paper at ICLR 2023
Lemma F.2. For the constants p (0, (()) and &L (0,1), given user U and its past records Pu,l-1, we suppose m, 7h, J satisfy the conditions in Theorem 4.2 and randomly draw the
parameter [O lt "
u IrT...... Consider the past records Putt up to round t are generated by a fixed policy when witness the candidate arms (Xr) Then, with probability at least 1 given an arm- arm-reward pair (It, we have
Σ (U(E)
T (z.: (1))) -rl [+]] V (V251+
3L
V2 +(++n)) +))
2oo
in a
where T, is the corresponding reward generated by the reward mapping function given an arm Ty-
Proof.
We proof this Lemma following similar approach as in Lemma C.1 from (Ban et al. 2022b)
and Lemma D.I from (Ban et al. 2022a). First, for the LHS and with 7 E Tu, U (t), we have
(a O (1) 1r) rzl S (
(1), ( (e ]+) Inel S 1 + 71
based on the conclusion from Lemma F.I Then, for user u, we define the following martingale difference sequence with regard the previous records Pu, up to round as
Vi-) B[[ (x+;: (6")) r-1] - 11 ( O 1+) - rll-
Since the records in set Pu,r are sharing the same reward mapping function, we have the expectation
((V(1) [url E|I (ar; 16_ }+) r- B (æri ( |+) Fr Fu,= = 0.
where Fu,r denotes the filtration given the past records Pu,r, And we have the mean value of v; across different time steps as
Σ
1 T
Σ TET.,*
E (x.; (r:; 1 1 r-] Tu,t TET..:
Σ (o.:
rzl-
Tutt rET...
with the expectation of zero. Then, we proceed to bound the expected estimation error of the exploitation model with the estimation error from existing samples following the Proposition from (Cesa-Bianchh) al., 2004). Applying the Azuma-Hoeffiie inequality, with constant 8 e (0, 1).
leads to
P Σ
E (
(x.i 10 1 -rel]
1
Σ TET..
11 rzl
Tu,t
Tu,t
TETu,
2 (1+m)
2
In( ) S.
1
V Tu,
As have the parameter (0@ @0) }}- f1 IrTTET with the probability at least 6, the expected loss on u could be bounded as
(1)
Σ (m;
_ (1) [+) - rell & (1 (1+m).
2
In( +
Σ
(2::
Twt TET...
V
Tu,t
Twut TETu,
where for the second term the RHS, we have
1 Σ
(
O (@")) 1+) r-1s Tu, TETu..
Σ ( (x++; 16 rrl
3L 2T
Tu, TeT..
2
Tu,
Tu, Σ (2, (1) r2
3L
Tu,
/ /TT,..
TeT...
E1
3L
V Tu,t
/2Tu,t
where the first inequality is the application of Lemma F.10, and the last inequality is due to Lemma [.6) Summing up all the components and applying the union bound for all a arms, all n users and time steps would complete the proof.
Then, also have the following Corollary for the rest of the candidate arms a., E (2/ (mx)).
24

---

Under review as a conference paper at ICLR 2023
U and its past records P,,-- we suppose m, M1, satisfy the conditions in Theorem 4.2 and randomly draw (1),
Corollary F.3. For the constants (0, (1)) and &1 (0, 1), given
the parameter [0 ~ {10) IrrTTT.... For an arm Lişt E X2, consider its union set with the the collection of arms Pu, U (ai,,; (i,s) are generated by a fixed policy when witness the candidate arms (XT)TETu.. with Pu,t = (z ri..rr being the collection of arms chosen by this policy. Then, with probability at least we have
3L Σ (O 10 - Ti., [ SV..t (VZE, TUT)
In (n-a)) Te m)y 2
+ +1
+1+
v2
where Ti, is the corresponding reward generated by the mapping function given an arm Bi,r, and
r (1+ L log" 5/6 (m))
t log(m)) + O I log 11/6 (m)
O
1/2
pm
n p4/3m
Proof. The proof of this Corollary follows an analogous approach in Lemma
First, suppose
shadow model (( (-: @ "6""" le), which is trained on the alternative trajectory P Pu,.- Analogous the
proof of Lemma F.2 for user u, we can define the following martingale difference sequence with
regard to the previous records P u,r up to round E [t] as
マリリ = El (a.,. O (1) (+) r-1] 15 (ii,.i
O (011)) Tir.l--
Since the records in set Pu.r are sharing the same reward mapping function, we have the expectation
B(V.")) Pu, Bl (a.. ( (1) (0)+1>>==))> ri..-ll B[ (z:., ( _ "1))) )-)
ri, Fu,r =
where Fur denotes the filtration given the past records Pu,r. The mean value of v.(2) across different time steps will
Σ 1
Σ B
(: @.
rl, Tu.t TETu..
Σ (2., --
Tw, TET...
Tu,t TETu.
with the expectation of zero. Afterwards, applying the Azuma-Hoefidiig inequality, with a constant 8 E (0, 1), leads to
P
Σ
(
(1) 1 1+) rir. Σ (Z :00 I+) ri..ll
Tu, TETu.
Tu, TETu..
2 V Tu,t
2 (1+m)
In S 6.
To bound the output difference between the shadow model (:: (6 (c) and the model we trained
based on received records f(() @ (1). Ic), apply the conclusion from Lemma G.14 which leads to that given the same input æ. we have
1 (m: 6 (1)
(1)
(a:; o 1+)
t log 5/6 (m) 1 [1+
LL log(m)) + o
( (t log (0001)(0(m)))
00
/
Pym
1/3
Finally, assembling all the components together will finish the proof.
L
F.2
USER EXPLORATION MODEL.
V { (x)
To ensure the unit length of ((()(()'s
(-)'s input, we normalize the gradient F9 and normalization constant dg > 0. Then, to satisfy the separateness
d with
Lemma
R88
Lemma
25

---

Under review as a conference paper at ICLR 2023
(Assumption 4.1
assumption, adopt the operation mentioned in Eq. to derive the transformation
", f (=;"o"))
66
x)) make sure the transformed input gradient is of the norm of 1, and the
separateness of at least
V2
Analogous to the user exploitation model, regarding the convergence result for FC networks in
Lemma F6 we proceed to present the generalization result of the user exploration model f!² after GD with the following lemma.
Lemma F.4. For the constants cg p E (0, (()) and & (0, 1), given user u E U and its past records Pu,--1, we suppose J1 satisfy the conditions in Theorem 4.2 and randomly draw the
parameter ( 1e {[O". 1r by a fixed policy when witness the candidate arms (Nr IrTT,, Then, with probability at least 1 given an arm-reward pair (æt, rt), we have
Irr..... Consider the past records Pu,e up round t generated
B B
V 160
fa (or: ( (1)
( -1-- (ar:::000)) 10(")))) |x
Σ
2r); #r);
Tg
rET,,1u())
(V
3L
(1 27)) log( tn aa
S SVuu
V2
Proof. The proof of this lemma is inspired by Lemma C. from (Ban et al. 2022b
Following the
same procedure in the proof of Lemma F.2 we bound
V 1.
(x): (1))
(re 1 (a)): (1)) )
((2) (
L
V
< ((2) Φf
-.):11))111))) + f(") (0!::
+1
< 2y1
by triangle inequality and applying the generalization result FC networks (Lemma on fa (:; O (1)
f(( (: (2)).
For brevity, use V (xx) to denote of( (a 1 (x:1 the difference sequence as
v.) for the following proof. Define
(
v VIL
(
(2)) rr
(x 10 ((()))))
10 (0))))))))))))))) -aar))-(r--- (r. (x.:(000))
((3))
(
Since the reward mapping is fixed given the specific user u, which means that the past rewards and the received arm-reward pairs (I,,, r7) are generated by the same reward mapping function, we have the expectation
( Fur =E
((2)
(2) "
(x (0
(1,, (r,- - 10
(r-- (()) (x); (0001))) |Farr = 0.
(2)
where Fu,r denotes the filtration given the past records Pu,r, up to round [4] This also gives the fact that vバ(2) is martingale difference sequence. Then, after applying the martingale difference sequence over Ta,ts we have
Σ 2
1
Σ
E
(2))
(2x)):
rt (+-- (() 1 (0(2)))) (+))
Tut TET
Tatt
TETT..
((2)
(x.
rr, - (x)) : (Q!
T TET...
26

---

Under review as a conference paper at ICLR 2023
Analogous to the proof of Lemma F.2
by applying the Azuma- Azuma- Hoeffding inequality, it leads to
log((1/)
(2)
Σ
2 (1 + 2
< 2
T
Tu.t
Tu
TETu,
Since the expectation of 1z(2) is zero, with the probability at least 1 and an existing set of parameters θ (2) (2)
| H
@3mm log m). the above inequality implies
s.t.
2 2lo((1/8)
Tw, Σ
V S (1 1
Tu,
TETU..
f(2) v u,
(
[0 [20)
((1) (x) 10
Tu,t TETui
< Ta, TETu.
((2) V (x));
f < (2:; O (-))) + (1 +(1+27)
21og(1 /6) Tu,
<
((0) V (x,);;
(a.);
θ (2)
- rr- 180 (1=-
(1 +1++21))
2 2lo((1/0) Tu,t
T u,t TETu,
(x - (r,- -0rr -1d (x.: + (1+ 2
2 log logg(1/8) (1/8) Tu,
((2)
Tu,e
TeTu
261 Tw.
21og(1/8)
<
+ (1 2n)
Tu,
(i)
Here, the upper bound (i) is derived by applying the conclusions of Lemma H.6 and Lemma F10 and the inequality (ii) is derived by adopting Lemma F.6 while defining the empirical loss to be
EreT..
(2)
V (z, (2)
rr
1 (x, (.()) Finally, applying the union
bound would give the aforementioned results.
Corollary F.5. For the constants (0, (()) and & (0, 1). given user U and its past records Pu,--1, we suppose m, Mh, J1 satisfy the conditions in Theorem 4.2 and randomly draw the parameter (0)), {10 IrTrTT... For arm aitt E X,, consider its union set with the the collection of arms Pu,, U (2i,, } are generated by a fixed policy when witness the candidate arms {X-)TETA with Pu,e = {Ki, di being the collection of arms chosen by this policy. Then, with probability at least
(1),
we have
V VioN
f(1)) (Ii (0))))
Σ B( TET),
2 1:
(0
ri., f(0) (i.,, x
Lg
(VDEi
3L (1+
tn
S TTat
+
2 log(
+ l.
V2
where Tir r is the corresponding reward generated by the mapping function given an arm vi,.r, and
1 log 5/6 (m) (
L
log(m)) + o LL log 11/6 11/6(m)
r = 1 O
O(
73333
nd pm
n 5 p/
This corollary is the direct application of Lemma E.4 and the proof is analogous to that of Corollary
27

---

Under review as a conference paper at ICLR 2023
F.3 LEMMAS FOR OVER- PARAMETERIZED USER NETWORKS
Applying Pu,e-1 as the training data, we have the following convergence result for the user exploita-
tion network ful (:; ()) after GD.
Lemma F.6 (Theorem from (Allen-Zhu et al. 2019)). For any 0 <1 11,0< O Given user u E U and its past records Pu,t-1, suppose m, ; satisfy the conditions Theorem 4.2 then with probability at least 1 - 6, we could have
1. () S E after J1 iterations of GD.
2. For any j E [Ji]. 1 10 O
(Te.)) pvm
log m
0
(n'p/m log m
In particular, Lemma F.6 above provides the convergence guarantee for fu 1 (: θ (1) after certain rounds of GD training on the past records Pu,--
Lemma F.7 (Lemma 4.1 in (Cao & Gu 2019 Assume a constant w such that Omm-3/22 L-2 (oggi// / I w < OIL llg m] -322) and n training samples. With
randomly initialized (0 for parameters O O' satisfying | we have
1 S w.
lf (x; - (x; 0') (Ve (2: a: 0)) 0')l 1 mm moggmm))l - e'll
with the probability at least - 5.
Lemma F.8. Assume m. m, J1 satisfy the conditions in Theorem 4.2
and _ lo being randomly
initialized. Then, with probability at least 5 and given an arm x 1, we have
1. fa (x: 1o)) 2.
2. V Y f (z: 1 (1))1))) O(L).
Proof. The conclusion (1) is direct application of Lemma 7.1 (Allen-Zhu et al. 2019 For conclusion (2), applying Lemma 7.3 in (Allen-Zhu et al. 2019), for each layer θ (O).. Or). we have
/ (x;
(DL [o)l2 = DL-1 ... 0 11+10 1 = =
Then, we could have the conclusion that
V le!") (x: 0 (+")))) 10)l22
V Ve,su
f (1) (a:;
( 1o) O(T).
[E(L)
Lemma F.9 (Theorem 5 in (Allen-Zhne Zhu et al. 2019)). Assume m. m, J1 satisfy the conditions in Theorem 4.2 and (O being randomly initialized. Then, with probability at least 1 - 5, and for all parameter (!)) such that ||O(!) ( (""llll I w, we have
V Ve fe ((1) (x; O (1))
V (a: (0)22 s
V
(e 1
133 /lg((m)
Lemma F.10. Assume m, mh satisfy the condition in Theorem 4.2
With the probability least 1 8.
we have
Σ (x,: -1) Σ TET..
(1) - -
1 rrl+
3L 3L 2Tu, /Tu.
TET..
Til lon(m)
Proof. With the notation from Lemma 4.3 in Cao & Gu 2019), set R
R2 and
LR TuTT.. would complete the proof.
Then, considering the loss function to be ((O!)) EFET.,, (()) (x:; (2))) rzl
28

---

Under review as a conference paper at ICLR 2023
G PROOF OF THE REGRET BOUND
In this section, we present the generalization results of GNN models fgnn (-; O Recall that up to round I, we have all the past arm-reward pairs P, {(r,, r> r>) for the previous time steps. Analogous the generalization analysis of user models in Section we adopt the the operation in Eq. on the gradients V Ve ann g @ to comply with
gmn). ggan( (-; θ (2)
gran..
the assumptions of unit-length and separateness, and the transformed gradient input denoted as A (x) given the arm
G.1 BOUNDING THE PARAMETER ESTIMATION ERROR
Regarding Eq given an arbitrary candidate arm e X₂ with its reward r, and its user graphs g(), g()), we have the bound for the estimation error
(B,(x) = E |
(a), g(a) (2) ganle-1) - (re- fgin(e (xx, g()) O (1) (())
us As
< E
[ggen (V ) ( (( O gunnle Je-)) - (r+ sgn (ar, g()), O gounn - })}
ueg Xi
+ + 1a + 14
where we have the term / representing the estimation error induced by the GNN model parameters {(0 g0nlle-1. (0 g Based on our arm selected strategy given in Algorithm we have the selected arms and their rewards (-,,rr) rrjtt-1 up to round t. And we first proceed to bound term
I] w.r.t. the selected arm æ,, i.e., (x+)..
Analogous to the user-specific models, we also have bounded outputs for the GNN models shown in the following lemma.
Lemma G.1. For the constants p E (0, O(t)) and & E 1), the past records Pa up to time step I, we suppose m, m, 12 J1, J2 satisfy the conditions in Theorem 4.2 Then, with probability at least
1 and given an arm-reward pair (x,r), r), we have
Aignnaa (z: 16 ggnn)) 92
where
12 2 (
L pym
+ + L244
(
p4/3m/1/ log 11/6 1/ (m)
log m
Proof. The proof of this lemma follows an analogous approach as in Lemma F. where we have proved the conclusion for the FC networks.
Given æ, denote the adjacency matrix of its estimated user graph (()) A (1) and have the normalized adjacency matrices as ((1)) (1) /n. For the received user u, U4 could deem the corresponding row of the matrix multiplication S X, represented by hu, = [S X Ja:, as the aggregated input for the network for the user-arm pair (æ, u.). Note that this way, the rest of the network could be regarded as L 1-layer FC network (one layer GNN L-layer FC network), where the weight matrix of the first layer is + (1)
agg" Then, to make sure each aggregated input has the norm A
of 1, we apply an additional transformation mentioned in Eq. as ha, o(hu, x)
( 2 Co,))
where Cu, = v information w.r.t. the user-arm pair (x, ut), as it does not change the original aggregated hidden representation. Meantime, this transformation also ensures the separateness of the transformed contexts to be at least G. which would fit the original data separateness assumption (Assumption 4.1. Finally, following similar approach in the FC networks (Lemma F.I. the transformed aggregated hidden representations would complete the proof.
hu, 12- This transformation ensures h ll 11 while preserving the original
29

---

Under review as a conference paper at ICLR 2023
Regarding the definition for the true reward mapping function in Section 2 we have the following lemma for term 11 given the arm-reward pair (at,, Nt).
Lemma G.2. For the constants p (0, O ())) and ε2 E (0, 1), given user u E l and its past records Pu,o, we suppose m, Th, 72, J1: J2 satisfy the conditions in Theorem 4.2 and randomly draw the parameters [6 ~ {[O (1) 1 - given a sampled arm-reward pair (2,rr), we have
gnnn=)-elt [9 19 ~ 1,6 (3) gnals T Then, with probability at least
1.6gmu
: 10 (piinl-1) (Te (a :100" (0 |1ve Xi g(
E
((2)
(1) (at):: (xt):
TETt]
3L (2222
(1 + 72) / 2 log( (n)-a)
S Vi.
v2
where
L Pym
L2
72 2
log m
+ +0
p4/3m-16 log (mm)
(m)
Proof. Based on the conclusion of Lemma
G.I we have the upper bound as
( g(2)'r;
| (pmil--) - Tt fgln (at,, g ((() ( (pinl--))) s 1 +1++2 202
by simply using the triangular inequality. Then we proceed to define the sequence
Vr, E [1]
V, =Ex,
Jan ( [T) (x.)) g( e (2) ppmnl -r-- - S (Z., g(1);; | (1) 1>)
p (V (( (x.),, g
gun 1--1) (r+ ffin (x,, g( O
(2)
(1) grn I- 1))
And since the candidate arms and the corresponding rewards are associated with the same reward mapping function l()),, the sequence V, is a martingale difference sequence with the expectation
EVV F+1 = Ex,
(VF. (1)* (x+),
G (2)+ 10 (2) (- - -r+- - - (xr, Gf
| (1)
gran 1-
- Ex:
(V)
(1)
(x
G
+ (2) ( - (rr - f(a (x,, g(U),,,, O (I)
gran J >>>
0.
where F- denotes the filtration of all the past records P, up to time step T. Then, we will have the mean value for this sequence as
V
rEL4
Σ Ex,
((2) gran
V ff(l))((,)) ; 1- - (rr fglli (x,,
(x g( [ ( ((unl=r)))
rE(d]
Σ rE((]
f(le) (V (1)+ (x+), ,. 1 (2) (- - (r, - -gpn (wr, g! 1
(1) 1 gnn ( 13)
As it has shown that the sequence a martingale difference sequence, by directly applying the Azuma- Ezuma-Hoeffding inequality, we could bound the difference between the mean and its expectation as
B 2 (1 + 272) 2 2log(1/8)
PP
V- 7E(1]
Σ FE(t]
< 5
t
30

---

Under review as a conference paper at ICLR 2023
with the probability at least - 23. Since it has shown that the V, is of zero expectation, have the second term on the LHS of the inequality to be zero. Then, the inequality above is equivalent to
2 (1/()
V+ S (1 + 272)
7E(d]
Σ
Ex,
((2)
V (x+),, g: : g (rr fggm (xr,, g ; (
(1) gunll-1))
7
gran
7E(4]
Σ
gran (V f(()), (x,), gf O () ( (r+ fgnn (E,, g( 1 gm00)mll111
TE((]
2 ll0(3
+ (1 + 2
t
with the probability at least - 28. Then, for the RHS of the above inequality, by further applying Lemma G.8 and Lemma G.12 we have
((2) gnn
(x.),, g(2);;; ( (2) gnn
I- (r+ sgnn (xT, g(t);;; ( gin - 1))
TE
1 Σ
(2)) (x+),
g gnn) (r- Sggn (2r, - [ 1pn 1))
(2)
7
3 2t
2
with regard to the parameter
gun s.t.
(2)
| gnn
(2)
1 gol2 0
13 log m
Therefore, by
Pvm
applying the conclusion from Lemma G.8 we could bound the empirical loss w.r.t.
gun as
7
Σ
((2) gnn
(V
1 (1), (x:), g( grn) - (rr - Sgann (xT,, g( [O(1] gopnnll 1))
=E((]
1
Σ
ggnn V (1)= (xx,, g( θ (2)
gnn)) (rr fggh (xr, gf ;1 ((00nrr-1))
2
vt
rE|d
26
Finally, assembling all the components and applying the union bound would complete the proof.
Analogous to the Lemma G.I we could also have the following corollary of the generalization
results for the optimal arms and their rewards {œ r=]rejd up to round t. Then, let 1 gan ] be the
(1),,,
parameter that is trained on (at,r;)reu and denote (@ (2),*, gnn }r the parameter of f!a) (-) trained
on corresponding gradients and residuals.
Corollary G.3. For the constants p E (0, ((+)) and &2 E (0,1), given user U and its past records Pu,to suppose m. M, 7,7 J1, Jz satisfy the conditions in Theorem 4.2 and randomly draw
the parameter ( (0)))))) {[6 gan ) re()). [ (0(2).11 { (2):*
(1),+
gnn }re(()- Then, with probability at least 1 - 5 given sampled arm-reward pair (I,r), we have
Σ B rE(e]
((2) gri
(wf). ((2),
:0 geinle-))
r
((1) fain (W -020==1000 (0thlle-)) |ue. Xe
(v25z +
3L
tn a 2log(
< Vt.
+ (1 + (1+
+ r
V2
5
31

---

Under review as a conference paper at ICLR 2023
where
(
L Pym
1,2
72 2
log
+ +0
log (0g11/0(m)) (m))
p//
Proof. The proof of this corollary is comparable to the proof of Lemma G.2 At each time step regarding the definition of the optimal arm, we have o max analogously, we could define the difference sequence as
E Fi, Me Ft,l]. Then,
V2 =Ex,
ggen
(x+),,
g(2),*
(2).a] gran
1-- (r+ "omm("; get): :
gran I --1))
f(e) gan
1-- (r+ - fgin( (zt g(
fu.n gun
1--1))
gun
where by reusing the notation, we denote  (I))* g(2)/* to be the true user graphs w.r.t. the optimal
arm x; here. Then, similar to the proof of Lemma G.2 we have the sequence to be the martingale difference sequence as
| (VI ( (xt), g(P))
B[V; [r:] = Ex,
(2),
- (1 (ar g(
(1).. gran
1)
gmn
- -Ex,
(V) (x)). g( gmn
(2).a gran 1-- (rr - fgin (at, g()); | (
gmn
with F. being the filtration of past optimal arms up to round r. Then, we could also applying the Hoefa-Hoe inequality to bound the difference between the mean 1 & V and its
expectation 1 E(g [[V:]. Finally, like in the proof of Lemma G.2 applying the conclusion from Lemma G.8 and Lemma G.12 would complete the proof.
Then, recall the definition of of the confidence bound function CB, w.r.t. the optimal arm af. we the corresponding term I as
I = E | (V (xx;,, o(2);; ; (0(22nle-11) - (r: (r-- --y f (2;,, g(") (0 pmn le- 11]] ue. Xe
And it can be further decomposed as
Ireen (V. (2) (zt),
(xt), G ((2),
(2) l- - (rr 192n (xi G (1)) 1 (1)
grn k- 1))l
I 1fgen
(xz), gl
2) Ie-1) - (r+- - fgln (xt g ()), 1
(1),
1)-+))
gnn
gnn
(xt g (1),;; gran
(1),*)
sghn (xi; (( gren -1)
+1
f (af).
(
|
( - fg2n V (mt), g( (2), gra
1
(2)
gun 1--1)
where the first term on the RHS could be bounded by Corollary G..3 Then, for the second term, we first denote h E R" to be the aggregated hidden representation w.r.t. the user-arm pair (u, æt) where u, is the i-th user. Here, h is essentially the row in the aggregated representation matrix H agg corresponding to the user arm pair (u,, x;)) Therefore, for the received user u, E U, the reward estimation based on two samples regarding the two sets of parameters would have the same the input hü- Then, for the second term, since the outputs w.r.t. two sets of parameters have the same input h;, could apply the conclusion from Lemma G.14 which will lead
| (a;; ((1).*; (O (1),
gmn ( - f (at; g (());*; (0()) (00()) I-
(1 (1+ O tL3 log 5/ (m) (m))
) O
LL
[L'2 1 (m)
[t log
log(m)) + O
p1/3m=/6
Pym
p4/3m=20 1/6
Analogously, we could also have the same bound for the third term on the RHS. Summing up the bounds for three terms on the RHS would finish deriving the upper bound for term JT.
32

---

Under review as a conference paper at ICLR 2023
G.2 BOUNDING THE EXPLOITATION GRAPH ESTIMATION ERROR
Then, we proceed to bound the error induced by the estimation of user exploitation graph, i.e., the emror term 22. Recall that the confidence bound function (ax) for the given arm X, is
(x) IE [ (Vft (z), 1
1 grale-1) (r- S (x,, g());
| gunnle
)))] eec Xs
S E gonn (xe, g());; O gomnl-11 - Lo (ae, f()) O (2) gnn 1- ues Xe
+11 + 13 + 11-
given an arbitrary arm Xr. For arm æ, we use the following lemma bound the error caused by the difference between the estimated exploitation graph ((1) and the true exploitation graph g())* associated with arm æ.
Denoting the adjaceney matrix of the estimated graph g()) A (1), and the adjacency matrix for the true user exploitation graph g(1),* as A (1),, we have the normalized adjacency matrices as ((2) = A A /n and ( A A /n. For the i-th user u, U, we could deem the i-th row of the matrix multiplication S Xx represented by ho, the network for the user-arm pair (x, u.). Note that in this way. the rest the network could be regarded as a L + 1-layer FC network, where the weight matrix for the first layer is ( Then, to make sure each aggregated input has the norm of 1, we apply an additional transformation
[S X]ss. as the aggregated input for
agg"
mentioned in Eq. as ho, o(ho, x) ( 2 Co.s) where co. / 1 And
this transformation ensures |ho.//2 |g- = 1 and co. 2 1.- Since this transformation does not alter the original aggregated representation i it will not impair the original information w.r.t the user-arm pair (at, Mz). Meantime, note that this transformation also ensures the separateness of the transformed contexts to be at least :-
Lemma G.4. For the constants p E (0, (0,0(+)) and & E (0, 1), given past records Pt-1, we suppose m, 1/1, 72, J1, J2 satisfy the conditions in Theorem 4.2 and randomly draw the parameter
[0(1) ganle {10 (1 bre(j)- Then, with probability at least
(1)
8, given
E R&, have
Σ E gnnn (æx, (( ( (0 sghn (x, (x, G. O (1)
rE(1
S SO(L) Vsi (v +
3L
in ) Pr. + (1+ +1++7) 2 log( 5
V2
Proof. By the conclusion of Lemma F2 time step t, the reward estimation enror of the user exploitation model could be bounded as
3L + +111 +(1+m)) Σ E. [ (z: (1 - -r [.] S / (V + V2 rE[t]
log( In (n-a) )) T +
with the probability least 8. And given two users u, uy E li and an arbitrary arm æ E Xi we denote their individual reward as r,, Ti separately. We omit the expectation notation below for simplicity. Then, we could bound the absolute difference between the reward estimations as
||r. -rr ry5 -15(') ( (O - (a,; [0 < [(r; - (0; (o))) - -r; - (0)
$ 1(r: f()) (z. (11) (ry 10-ff (0
((r, f O ()) 16r (Z)
Based on the definition of the mapping function v, it would naturally be Lipschitz continuous with the coefficient of 1, which is [exp(-Irz ryl) exp(-15(() (x;; @1)) ( (0
||r - (c
O @1)) fl (a,:10@[1ll
33

---

Under review as a conference paper at ICLR 2023
with the probability least - 8. Finally, applying the union bound for all the (n² n2 user pairs and re-scaling the would give us the estimation error bound for the reward difference for each pair of users. To achieve the upper bound, apply the Corollary F3 by considering the trajectory Pu,t consists of the past arm-reward pairs (Zi,,r, (r(()) where
Bi,,7 leads to the largest estimation error of the estimation model sar(()) in each round r E [t]- Thus, we have the bound for the edge weight difference, where the difference of an arbitrary i-th row could be bounded by
Σ [1(4 - (A "1llla S 2nvi. (V25 +
3L
2 log( tn x)) 8
+ (1 + +
+ re>
V2
TEA
which implies
Σ E1lss - -ss 22v (v +
3L v2
tn a ()) Ir.
+ (1 + m logg(
rE(1
Therefore, applying the conclusions from Lemma
F2 it leads to
Σ rE(t]
||r rarl | (11
(.),,; ( 10
3L 2 V _.. (V25)
V2 (1 + m) 2 tn in.a.
+ rr
Afterwards, recalling the transformation at the beginning of this subsection, and given an user-arm pair (w,, 2) for the i-th user, we denote h = [S(1) X]: and h. = [g(1),* X]:: Based the aforementioned transformation in Eq. their transformed form could naturally be h ( ( h c)
and h - (> h" 2 c²) with ||>||| 1. Without the loss of generality, we let c c*. Then, could have
1 h. 112 = V h 221 + (c - c²
< v lh (1)
h 1 (c2 (c)))
11 - h 11113
(1)h² 112 -1/2113)2
4
1 (111" -1ll2 112 + h2ll2)²
Ih h 112
/2- || h²' h²ll2
(iI)
Here, (i) is because c* 2 (ii) is because of c² + 11:13 = (c+)2 + I1k'
2 4. and (iii) is due to
|/l||,, ||h]|e 1l2 1
Then, we proceed bound |/2 h" h"ll Recall the definition from Eq. conclusion across different rounds T E [t], we will have
Extending the above
Σ 1h, h;lla < ||+||| ||[S
<2 2E, +
3L V2
+ (1 + n))) 2 In. (n))))))))))) + r..
TE(
Finally, combining the conclusion from Lemma G.13 we finally have
Σ 1 (x, g(",;; ( f(ll) (x, g()) O (1) --
TE(
O(L V8L (V25) + 3L
on a (oe
+ (1 + m) 2log(
re
V2
which concludes the proof.
34

---

Under review as a conference paper at ICLR 2023
G.3 BOUNDING THE EXPLORATION GRAPH ESTIMATION ERROR
Again, recall the definition of the confidence bound function ( which is
(B, IE g V (x), g( [0 1ognnnn-1 (r -ggnn (æ, g(); O ( grnle- - ))
ees Zts
S E ggan (1) (z), g (2), (x),
(2) gon
l--1) sgmn (V (1) (æ), g O
( yun
ل--1) dee Xi
Is
+11 12 + 4.
Analogous to the procedure for the user exploitation graph, we have the following lemma to bound the error induced by user exploitation graph estimation.
Lemma G.5. For the constants E (0, (())) and & E (0, 1), given past records Pi-1,
suppose m, 1/1 72. J1, J2 satisfy the conditions in Theorem 4.2 and randomly draw the parameter
[O (2) gnnne ~ { 1 gnn limllreete ]r(((-- Then, with probability at least
(2)
6, given an arm x E R we have
gnn 11)+ (x), g( 1 (2) (onn f(2) V (x)())) g + pmnl-11)
TE
$ OO(L) Vst. (v251 +
3L
log In =)) 8 + Tee
V2 + (1 + (1+
Proof. The proof of this lemma could be derived based on a similar approach as in Lemma G.4 Recall that for the exploration GNN model 19mn( (-), we have the gradients of the GNN exploitation
[", (m:(())
( (x)
model V
the input given an arm æ and user U, whose norm
IV (m)]l2 S 1.
Given two users u,, uy E l and an arbitrary arm æ E Xr we denote their individual reward as Ti, T separately. Then, could bound the absolute difference between the potential gain estimations
I1(r: f( 10) (ry f( (2; [ ((())))) 1sr
11)
(V f (z): (9)1))
< | (x); 0i (+)) - (n- 1 (x; O [+))
+ + (V (x): (z): (ry f (z; + (1)))))
Afterwards, applying the conclusion from Lemma F44 would lead to the result that
Σ (1(r: - f( ( (11))) - (ry - (x: [O (o11)) - 1 (V (1) (a)
())))))))))) (a)
(a))
(z); (=)) [1
TE(#]
$2/ - 225 V
3L + +1+ +1+ log(
In +
8
Following a similar approach as the proof of Lemma hidden representations for the input gradients. Since the entries between A (2) A (2),* (and also
G.4 we proceed to consider the aggregated
the distance between 5(2) (()).*) are bounded, by adopting the aforementioned transformation in Eq. on the aggregated hidden representations for the input gradients and the initial arm contexts æ, we would end up with the bound for the difference between transformed representations for input gradients. Finally, combining the conclusion from Lemma G.13 would give the proof.
G.4 BOUNDING THE GRADIENT INPUT ESTIMATION ERROR
For the last term I in the confidence bound function CB,(),, we have
4 = ( (VJ (1), (x)) g(2); 1 (2)
m ))- f(2) (V (1) (æ:), g(2); 1 (2)
(0 ( ur X
35

---

Under review as a conference paper at ICLR 2023
which represents the estimation enror induced by the difference of input gradients. And we first bound the gradient difference with the following lemma
Lemma G.6. For the constants E (0, (()) and & E (0, 1), given past records PE-1, we suppose m, Mu, 72, J1, J2 satisfy the conditions in Theorem 4.2 and randomly draw the parameter
(1)
grant
{ gua! 3-]((- Then, with probability least
6, given x E TRd. we have
V (x) v. (2)1l2
rE(d]
( 1 log 5/6 (m)) (LL) v8t. (V251+ +1
log
3L
+ 2 In (n) T₁
50 p2/3m2/6
V2
((1) Ren
(1), (a, (0 and V (x)
Vo Jonn (e, gev). 1 ( onn
where (1((
Proof. Following the aggregation procedure and transformation procedure shown in section G.2 have the transformed representations for given an user-arm pair (us, x) with the i-th user, which are h = [S(1. X] and h² = [s(1), X].. And their transformed form could naturally be h (h, c)
and h (h" c²)) From the conclusion of Lemma G.4 have
Elh, - h2 S Vst V
(V2 +
3L + (1 + 2 log (n)0)) Ir. v2
TEjO
Then, applying the conclusion from Lemma G.13 would complete the proof.
Then, we have te following lemma to bound the term 1.
Lemma G.7. For the constants p E (0, O))) and & e (0, 1), given past records Pt-1, we
suppose M1, Tp. J1, satisfy the conditions in Theorem 4.2 and randomly draw the parameter
[!1l] gankt
{ 1 ( Then, with probability at least
(1)
8, given an arm æ E Rd, we have
Σ p (()) (x:), g(2); (0)2
(2) ( ((2) (V (()) (x), (2)) gnn 1-1
(2)
TE[t]
12 log
3L log 10 ([I2) v8t (v25i
+ (1 m)) 2log( in .0 /2
5 ))
s 20 1333
+ De-
Proof. We again follow the aggregation procedure and transformation procedure presented in section G.2 Then, the aggregated and transformed input gradient could be denoted as we have the transtormed representations for given an user-arm pair (u; with the i-th user, which are [s( GGg and g" = [s(?) G"k.. where G denotes the gradient matrix embedded w.r.t Eq. 4 And their transformed form could be ğ 222 g: e) and g = (>22 g" e*). Then, according the definition of Eq. 4 we could naturally have
و 12 < |I(S
|
(x) V (f (x)lla < IV (x) (
v (z)
since the normalization of the adjacency matrix ensures its arbitrary row has the norm smaller than 1. Finally, applying the conclusions from Lemma G.6 and Lemma G.13 will leads to
V (x), g (2); (0 -- - f(2), ( (x), g(2);
(2) --
gan
TE(1]
3L 2 L 12g (m) O(L3) v8t. (vZzi+
tn .)) + -r,
00 f133m 1/6
+ (1 + 7
2 log(
V2
36

---

Under review as a conference paper at ICLR 2023
G.5 LEMMAS FOR OVER- PARAMETERIZED NETWORKS
Applying Pa-1 as the training data, we have the following convergence result for the exploitation
GNN network fu (1; + (1) after GD.
Lemma G.8 (Theorem 1 from (Allen-Zhu et al.
2019).. ) For any 0 < & < O
Given past records Pt- suppose m, 72; J J2 saltsfy the conditions in Theorem 4.2 then with probability at least - 6, we could have
1. ((O@)) gmn
&2 after J2 iterations of GD.
2. For any j E [J2] 119 gmm. - [ (1)
1911 S S0(
log m
grin
Pvm
In particular, Lemma F.6 above provides the convergence guarantee for fo (:; e gnn) after certain rounds of GD training on the past records P--1-
Lemma G.9 (Lemma 4.1 in (Cao Gu 2019)). Assume
constant w such that
O(m (ooooo/ /o))) I w < O "log m] 3/2) and n training samples. With
randomly initialized (0(1) we have
gonlo. lo. for parameters O O' sarisfying
e - [o gonloll. | jo( gnnlol S w.
(z: 0) 0') (Ver (()) (x; 0²,, 0() O Vm log 0'l
with the probability at least - 5.
Lemma G.10. Assume m, Th 72, J1. J2 satisfy the conditions in Theorem 4.2 and [ (1) being randomly initialized. Then, with probability at least ê and given an arm a 2 = 1, we have
1.
(1) (x:
(x:;
(1)
gmn 10)l 2
2.
V 1 aanjo
(
1o) < O(L).
Proof. The conclusion (1) is direct application of Lemma 7.1 (Allen-Zhu et al. 2019). For
conclusion (2), for each weight matrix θ (e(() (1) (1)
( ( where O (I) = (1)
agg we have
I ( (x;
gmn 1o)ll = |10,, OD D.0t:i......00... D D. S O(VL)
(1)
by applying Lemma 7.3 in (Allen-Zhu et al. 2019), and denotes the aggregated hidden representa- tion for each user-pair, namely the corresponding row in H agg Therefore, by combining the bounds for all the weight matrices, we could have
p (1) (() (a:; (1(1) 12)
Σ V (E(O.....)))
Iool (z; (1) (o)n O(L))
grvee Jo
which finishes the proof.
Lemma G.11 (Theorem 5 in (Allen-Zhu et al. 2019)). Assume the training parameters m, (2,712, J2 ( gualo being randomly initialized. Then, with probability
satisfy the conditions in Theorem 4.2 and O
least
8, andfor all parameter t (1)
gron such that
||@ (1) (1) @ 2 I w, we have
grin
| e (2; (1) (1))))
g ( (x:; (1( 1o S (Vogg)))
Lemma G. Assume m. 1/2 satisfy the condition in Theorem
4.2
With the probability at least
1 6, we have
E1(( 1 - +115
-
Σ (
(1) ( (e))e) rrl
3L 2
7E(1]
rE(t]
37

---

Under review as a conference paper at ICLR 2023
Proof.
With the notation from Lemma 4.3 in (Cao Gu 2.19), set R / (lo((m)
R², and
8 Te(l ( O gnn) Trl would
LR Then, considering the loss function to be L(O) /2vt
[T)
gran)
complete the proof.
Lemma G.13. Consider L-layer fully-connected network f(;; 22) initialized w.r.f. Subsection
3.2.1 For any 0 < &2 S 1, 1.0 O(E). Given the training data set with t samples satisfying the
wnit-length and the p-separateness assumption, suppose the training parameters m, 122 J2 satisfy the
conditions in Theorem 4.2 Then, with probability at least 1 - 8, we have
[f(: O2 f(x'; 0. - OLL) || x'|)
[ax; ).) - Ve, 02) S O LL log 3/6 (m)
+ O(L) || 2'11
p2/3m226
when given two new samples x,
Proof. Denoting Di to be the diagonal sign matrix of the 1-th layer such that Di(i, 1((0//:11); 2 0j E [m], we could have
(s; O.) [(a'; 0.)1 1(0), ... D1O1) (ar x')'|
< 110 ... D | x'1l2.
Based Lemma 7.3 from Zhl al. 2019) and Lemma C.4 from (Ban et al. 2022b). O(L) for the initialized parameters Ou {OO) 0: ..... θ (.)o)- Meantime, after training the network and ending up with trained parame- ters θ, | [Or],], according to Lemma 8.6 from (Allen-Zhu et al. 2019), the bound OLL) still holds, which proves this statement.
we have we have |0, 1
Then, for the bound on the gradients, we have
[V, 0,) f(x'
|Ve, 0.) Ve, 0n) + (a, (e, + Ve,' Ve. fe. + Ve. Ve.f(='; 0,)
/e. ;
Firstly, we have
[[= = - D1+1
01- (14 21/ - O(VL)
based on Lemma 7.3 from (Allen-Zhu al. 2019), and this leads [e: s O(L). Analogously, we also derive
Ve,,f(x: Op) Ve.f(z'; 0n)
-I Di+ 0 (D 0 D201) (z- z') - O(L) || x
Then, according to Theorem 5 from (Allen-Zhu et al.
2019 and with | 111|2 S w, we could have (x; 2o) Ve, 0.] < O w 1/3 (o]] /V,, θn ||22 Substituting
the value with the conclusion from Lemma G.8 could have
Ve.f(x; O₂) - ve.f(z;;,)) l2 S ( 12 Voo(llog)))))))))))))mm))))))))) /Ve,f 1,)12
tL log 5/6 (6())
log
0 1/3
Finally, assembling all parts together will lead to the conclusion.
Lemma G.14. Consider L-layer fully-connected network f(:; Ot) initialized w.r.t. Section 3.2.1 For any < & < 1, < p S O(t)- Let there be two sets of training samples Pt, Pt with the unit-length and the p-separateness assumption, and let O be the trained parameter P. while
38

---

Under review as a conference paper at ICLR 2023
Ø is the trained parameter on Pl. Suppose m, 12. J1, J2 satisfy the conditions in Theorem 4.2
Then, with probability at least 6, we have
If(:; = 01 -
1+ 1 LL log 5/0 1og"/((1)
00 log(m)) 0
LL
(m)) (0
(11 log loggl///10 (m)
pl/3m 1/6
PVm
p//3m1/6 pp/mmm166
when given new sample a E Rd.
Proof. First, based on the conclusion from Theorem 1 from (Allen Zhu al. 2019) and regarding
the samples, the trained the parameters satisfy || | @ul2. | Ooll I O pvm log(m)) = w
where O is the randomly initialized parameter. Then, we could have
+ [ 0.) S f O f(V O,)
1 O0 tL3 log 5/6 (m)
log
(m)) ) O(L) p2/3m 1/6
w.r.t the conclusion from Theorem and Theorem 5 of (Allen-Zhu et al. 2019). Then, regarding the Lemma 4.1 from (Cao & Gu 2019), we would have
If(z; θ2) f(:; 0%) (a; O²), O. Ø')l s
||0, -6/12
Therefore, the target could be reformed
fx ,.) f(x; 0') & [[Ve; 0 - 0/112 + 1(w1/3 m mog(mm) |0, - 0.112
1 0 pl/3m1/6 (mm))) O(L) w o( / logmm)
113 1 10g (m)
Substituting the with its value would complete the proof.
H COMPUTATIONAL RESOURCES
All the experiments are conducted on a Windows machine with an Intel Core i7 CPU, 64GB RAM, and two RTX 5000 GPUs.
39


Reviews:
[{"review_id": "p7vuxbaInCJ", "reviewer": "Reviewer_ENq3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a novel graph neural bandit approach to tackle the problem of estimating the reward corresponding to the selection of options for users based on their contextual information. Differently from previous works, they adopt a graph structure among users to exploit the correlations existing among them. The authors propose and analyse a new algorithm theoretically. Finally, they apply their methods to real-world settings and show the superior performance of what has been presented w.r.t. state-of-the-art algorithms.", "summary_of_the_review": "The topic is interesting, and the developed approach is novel, but some concerns about the presentation and self-completeness of the main paper make it not yet ready for publication.", "strength_and_weaknesses": "The topic and the methodology proposed are for sure of value to the ICLR community. I have concerns mainly about two points.\n1) The presentation of the paper is poor. Before considering it for publication, I think that the authors should revise the structure of the paper and to better present the elements composing the proposed approach.\n2) The paper is not self-contained. Indeed, while it is clear that the proofs cannot be included in the main paper, most of the details about the experimental setting are left in the appendix. This also holds for some parts of the description of the methodology (pseudocode). My concern is that the paper requires the appendix to be fully understood.\n\nMoreover, I think that the authors should provide statistical significance for the experiments they provided.\n\nDetails:\n- please add a formal definition of what you call volume (before Equation 1)\n- check the punctuation of the formulas \n- please add a more complete definition of the \\Lambda^*_i,t matrix. For instance, is it changing over time?\n- The presentation of the user graphs is not clear. In my opinion, you should have presented a general definition for a graph and subsequently instantiate it for your exploitation and exploration graphs. I suggest rethinking this paragraph.\n- Is the function \\Psi^{(1)} known? It is mentioned in the following part of the paper, but it should require a discussion just after its definition. The same holds with \\Psi^{(2)}.\n- there are minor grammar errors (in -> at, missing 'the')\n- Overall, the presentation of the problem formulation can be improved. I think that all the elements required for further discussion are present, but the order and the comments provided may be unclear to a generic ML researcher who is reading the paper, while an expert of the bandit setting is able to understand even if they are presented poorly.\n- Remark 3.1: do we also have some bounds on the loss we might incur in terms of accuracy or the bias we are introducing with this approach?\n- Remark 3.2: even in this case, it might be interesting to understand if there is a factor in terms of regret implied by this choice.\n- I think that deferring the details about the algorithm to the appendix would make the paper non-self-contained. Maybe the complexity of the description of the proposed algorithm requires more space to include all the material, therefore, I suggest evaluating the possibility of sending the paper to a venue where there is no constraint on the page length.\n- It seems that the experiments are provided without confidence intervals/standard deviation bounds. I think that the significance of the results can be assessed only if they are explicitly added to the figures.\n- The details about the experimental setting are provided in the appendix. I think that the authors should also distribute the code corresponding to their method.\n\n------------------------------------------------------------------------------\nAfter rebuttal: I think that the paper is of interest but should be accepted only if the authors are able to make it self-contained. I raised my score, assuming that the authors can implement all the improvements cited in the rebuttal.\n\nAfter the second rebuttal response: I think that the quality of the paper presentation has been improved during the discussion. I raised further my score.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mOiyfrh8ep", "reviewer": "Reviewer_WB9s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new model called Graph Neural Bandits (GNB) to leverage the collaborative nature among users empowered by graph neural networks (GNNs). Instead of estimating clusters of users, the method models the “fine-grained” collaborative effects through estimated user graphs in terms of exploitation and exploration individually. Based on EE networks, the paper utilizes separate GNN-based models for exploitation and adaptive exploration respectively. Theoretical analysis and experimental results on multiple real data sets in comparison with state-of-the-art baselines are provided to demonstrate the effectiveness of the proposed framework. ", "summary_of_the_review": "see detailed Strength And Weaknesses section.", "strength_and_weaknesses": "Although the motivation of this paper is well and the idea of the proposed framework is well defined, However, I have the following concerns about its learning process, running and space complexity. \n\nI feel that its complexity seems to be very high, which will hinder its real application. \n1.\tThis paper tries to model the fine-grained collaborative effects of individual users and the arms. This motivation is straightforward, however, it may make the scale of the network be larger significantly. In a real online recommendation system, there would be several billions of requests from millions of different users. These users are evolving as the time. If we try to model the individual user in the bandit setting, the network should be larger extremely, which may highlight the sparsity problem of the data. Is it necessary to model the individual user after taken the large n into consideration? On the other hand, if we have a small number of users for an application scenario, the data may enough for the exploitation. In this situation, the advantage of exploration is relatively limited. \n2.\tIn the learning framework, the method will construct two graphs continuously. The space and running time complexity should be considered. I feel the complexity would be larger significantly to obtain the fine-grained reward improvement. So I suggest to compare the complexity except cumulative reward. \n3.\tAlthough the learning framework is clear, the convergence of the exploration and exploitation GNN should be given. These two GNN are dependent. Does the structure of exploitation gnn have effect on the convergence of exploration gnn and vice versa? Will the prediction of the exploration gnn be smaller as the increase of the data? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YWosSLB5MQ", "reviewer": "Reviewer_hbM8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel framework named Graph Neural Bandits (GNB) to model the fine-grained user collaborative effects where they model the user correlation by the user graph. A GNN-based model is proposed to achieve adaptive exploration. They also show an O(\\sqrt{T\\log(Tn)}) regret upper bound of the GNB algorithm. Extensive experiments show the efficiency of GNB compared with state-of-the-art algorithms.", "summary_of_the_review": "This paper proposes a novel framework named GNB to model fine-grained user collaboration. It also gives a sharper regret bound of the GNB algorithm compared with existing results. It will be better if the author could give more discussions about the experiments. ", "strength_and_weaknesses": "Strength:\n\n1. The idea of using the user graph to model the fine-grained user collaborative effects is new. The proposed framework GNB is also novel.\n\n2. The regret bound illustrated in this paper is sharper than the existing results. They remove the term $d$ and reduce $\\sqrt{n}$ to $\\sqrt{\\log(n)}$.\n\nWeakness:\n\n1. I’m confused about the term $d$ in the regret bound. Can you give more explanation about how to remove the term $d$ in the analysis?\n\n2. The regret theorem requires $m > \\Omega(poly(T))$, which might be very large when round $T$ is sufficiently large.\n\n3. Can you give more explanations about the exploration user graph? How about estimating the potential gain based only on a single user and arm?\n\n4. The experimental results in the MovieLens dataset and Yelp dataset seem not to converge. Maybe the algorithm needs to run more rounds to show the convergence.\n\n5. It will be better if the author could discuss more the comparison between the linear algorithm and neural architecture algorithm, and the comparison between the algorithm considering user collaboration and the algorithm considering no user collaboration. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "YQJW7dd434o", "reviewer": "Reviewer_tRgB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary\n\nThis paper introduces Graph Neural Bandits (GNB) in the context of personalized recommendation. Specifically, the authors leverage graph neural networks to learn users’ “coarse-grained” collaborative instead of “fine-grained” for bandits optimization. All in all, the proposed method shows good performance across nine algorithms/baselines on various datasets and the authors also provide the theoretical analysis for regret upper bound of complexity.\n", "summary_of_the_review": "Overall, I am personally a bit concerned about the paper’s novelty (i.e., contribution). The authors did great job about the experiments and theoretical analyses, but some minor parts could be improved. \n\n=====\n\nIncrease my score from 3 to 6 after the rebuttal", "strength_and_weaknesses": "Some strengths:\n\n1. The paper tackles an interesting bandits problem by formulating the exploitation and exploration via graph neural networks. The paper has extensive experiments.\n2. The proposed method seems to perform consistently across datasets according to Section 5 Experiments and Appendix B. \n3. The paper is straightforward and easy to follow\n4. It’s also good that the authors provided source code for reproducibility.\n\n\nSome of my concerns/suggestions for improvements: \n\n1. In my opinion, the contribution is limited as the paper is developed based on [1, 2], which made the contribution is incremental. It seems to me that the contribution is narrow down to instead of using exploitation and exploration neural networks [1], the authors constructed exploitation and exploration graph neural networks (GNNs) to solve the problem. Specifically, those graphs are weighed graphs in which each edge weight is an exploitation/exploration score. \n2. Figure 2 shows cumulative regrets on four datasets: two from recommendation and two from classification. While they all show good results, it’s not trivial to understand why the ‘curves’ have much difference for the two recommendation datasets (i.e., almost linear) compared to the two classification datasets. Section 5.2 should give more detailed explanation and analyses. Why this method focuses particularly to personalized recommendation? Can it be generalized to other tasks?\n3. While the authors provided multiple remarks and theoretical analyses on the complexity, I believe it would be better if the authors could also provide the run time of GNB and all the baselines. This is for us to have an overview if we should use the proposed method, or a single-bandit setting such as [1] is good enough, given that they all have similar bound. Moreover, I believe it could also give the readers a benchmark given the resources in Appendix I.\n4. Figure 7 (Cumulative regrets for different exploration coefficients alpha) and Figure 8 (Figure 8: Cumulative regrets for different neighborhood hops k) show similar results. Thus, should we also study the relationships between alpha and k? \n5. Also, according to Figure 8 and Appendix B.7, learning with 1-hop is good enough, and larger k (k=2,3) could potentially lead to over-smoothing problem. With this observation, it leads to a question that do we really need to build graphs for exploitation and exploration dilemma in the context of personalized recommendation? Building large-scale graphs in practice are also expensive even if we can use user neighbourhood approximation such as Appendix B.4. Please give more explanation/discussion.\n\nMinor:\n\n1. I think the title should be more specific towards recommendation instead of only “Graph Neural Bandits”\n2. Appendix H Technical Lemmas is redundant in my opinion\n\n\nOverall, I am personally a bit concerned about the paper’s novelty (i.e., contribution). \n\n[1] Ee-net: Exploitation-exploration neural networks in contextual bandits. ICLR 2022.\n\n[2] Neural collaborative filtering bandits via meta learning. ArXiv 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject