Title: PageRank Bandits for Link Prediction
Venue: NeurIPS 2024
Accepted: yes
Keywords: Link Prediction

PageRank

Graph Mining
Abstract: Link prediction is a critical problem in graph learning with broad applications such as recommender systems and knowledge graph completion. Numerous research efforts have been directed at solving this problem, including approaches based on similarity metrics and Graph Neural Networks (GNN). However, most existing solutions are still rooted in conventional supervised learning, which makes it challenging to adapt over time to changing customer interests and to address the inherent dilemma of exploitation versus exploration in link prediction.
To tackle these challenges, this paper reformulates link prediction as a sequential decision-making process, where each link prediction interaction occurs sequentially. We propose a novel fusion algorithm, PRB (PageRank Bandits), which is the first to combine contextual bandits with PageRank for collaborative exploitation and exploration. We also introduce a new reward formulation and provide a theoretical performance guarantee for PRB. Finally, we extensively evaluate PRB in both online and offline settings, comparing it with bandit-based and graph-based methods. The empirical success of PRB demonstrates the value of the proposed fusion approach. Our code is released at https://github.com/jiaruzouu/PRB.

PageRank Bandits for Link Prediction

Abstract

Link prediction is a critical problem in graph learning with broad applications such as recommender systems and knowledge graph completion. Numerous research efforts have been directed at solving this problem, including approaches based on similarity metrics and Graph Neural Networks (GNN). However, most existing solutions are still rooted in conventional supervised learning, which makes it challenging to adapt over time to changing customer interests and to address the inherent dilemma of exploitation versus exploration in link prediction. To tackle these challenges, this paper reformulates link prediction as a sequential decision-making process, where each link prediction interaction occurs sequentially. We propose a novel fusion algorithm, PRB (PageRank Bandits), which is the first to combine contextual bandits with PageRank for collaborative exploitation and exploration. We also introduce a new reward formulation and provide a theoretical performance guarantee for PRB. Finally, we extensively evaluate PRB in both online and offline settings, comparing it with bandit-based and graph-based methods. The empirical success of PRB demonstrates the value of the proposed fusion approach. Our code is released at https://github.com/jiaruzouu/PRB

1 Introduction

Link prediction is an essential problem in graph machine learning, focusing on predicting whether a link will exist between two nodes. Given the ubiquitous graph data in real-world applications, link prediction has become a powerful tool in domains such as recommender systems [72] and knowledge graph completion [49, 41]. Considerable research efforts have been dedicated to solving this problem. One type of classic research approaches is heuristic-based methods, which infer the likelihood of links based on node similarity metrics [43, 46]. Graph Neural Networks (GNNs) have been widely utilized for link prediction. For example, Graph Autoencoders leverage Message Passing Neural Network (MPNN) representations to predict links [29]. Recently, MPNNs have been combined with structural features to better explore pairwise relations between target nodes [73, 70, 18, 61].

Existing supervised-learning-based methods for link prediction are designed for either the static [73, 70, 18, 61] or relatively dynamic environment [64, 55, 62, 58, 69, 19, 27, 26, 75], they (chronologically) split the dataset into training and testing sets. Due to the dynamic and evolving nature of many real-world graphs, ideal link prediction methods should adapt over time to consistently meet the contexts and goals of the serving nodes. For instance, in short-video recommender systems, both video content and user preferences change dynamically over time [28]. Another significant challenge is the dilemma of exploitation and exploration in link prediction. The learner must not only exploit past collected data to predict links with high likelihood but also explore lower-confidence target nodes to acquire new knowledge for long-term benefits. For example, in social recommendations, it

is necessary to prioritize popular users by 'exploiting' knowledge gained from previous interactions, while also'exploring' potential value from new or under-explored users to seek long-term benefits [7]. Furthermore, while existing works often analyze time and space complexity, they generally lack theoretical guarantees regarding the performance of link prediction. To address these challenges, in this paper, we make the following contributions:

Problem Formulation and Algorithm . We formulate the task of link prediction as sequential decision-making under the framework of contextual bandits, where each interaction of link prediction is regarded as one round of decision-making. We introduce a pseudo-regret metric to evaluate the performance of this decision process. More specifically, we propose a fusion algorithm named PRB (PageRank Bandits), which combines the exploitation and exploration balance of contextual bandits with the graph structure utilization of PageRank [59, 42]. Compared to contextual bandit approaches, PRB leverages graph connectivity for an aggregated representation. In contrast to PageRank, it incorporates the principles of exploitation and exploration from contextual bandits to achieve a collaborative trade-off. Additionally, we extend PRB to node classification by introducing a novel transformation from node classification to link prediction, thereby broadening the applicability of PRB.

Theoretical Analysis . We introduce a new formulation of the reward function to represent the mapping from both node contexts and graph connectivity to the reward. We provide one theoretical guarantee for the link prediction performance of the proposed algorithm, demonstrating that the cumulative regret induced by PRB can grow sub-linearly with respect to the number of rounds. This regret upper bound also provides insights into the relationship between the reward and damping factor, as well as the required realization complexity of the neural function class.

Empirical Evaluation . We extensively evaluate PRB in two mainstream settings. (1) Online Link Prediction. In this setting, each link prediction is made sequentially. In each round, given a serving node, the model is required to choose one target node that has the highest likelihood of forming a link with the serving node. The model then observes feedback and performs corresponding optimizations. The goal is to minimize regret over T rounds (e.g., T = 10 , 000 ). We compare PRB with state-ofthe-art (SOTA) bandit-based approaches (e.g., [76, 12]), which are designed for sequential decisionmaking. PRB significantly outperforms these bandit-based baselines, demonstrating the success of fusing contextual bandits with PageRank for collaborative exploitation and exploration. (2) Offline Link Prediction. In this setting, both training and testing data are provided, following the typical supervised learning process. Although PRB is designed for online learning, it can be directly applied to offline learning on the training data. We then use the trained model to perform link prediction on the testing data, comparing it with SOTA GNNs-based methods (e.g., [18, 61]). The superior performance of PRB indicates that principled exploitation and exploration can break the performance bottleneck in link prediction. Additionally, we conduct ablation and sensitivity studies for a comprehensive evaluation of PRB.

2 Related Work

Contextual Bandits . The first line of works studies the linear reward assumption, typically calculated using ridge regression [39, 8, 1, 60, 21, 53]. Linear UCB-based bandit algorithms [1, 9, 40] and linear Thompson Sampling [4, 2] can achieve satisfactory performance and a near-optimal regret bound of ˜ O ( √ T ) . To learn general reward functions, deep neural networks have been adapted to bandits in various ways [10, 11]. [54, 47] develop L -layer DNNs to learn arm embeddings and apply Thompson Sampling on the final layer for exploration. [76] introduced the first provable neural-based contextual bandit algorithm with a UCB exploration strategy, and [74] later extended to the TS framework. [22] provides sharper regret upper bound for neural bandits with neural online regression. Their regret analysis builds on recent advances in the convergence theory of over-parameterized neural networks [24, 5] and uses the Neural Tangent Kernel [34, 6] to establish connections with linear contextual bandits [1]. [12, 13] retains the powerful representation ability of neural networks to learn the reward function while using another neural network for exploration. [52, 51] integrates exploitation-exploration neural networks into the graph neural networks for fine-grained exploration and exploration. Recently, neural bandits have been adapted to solve other learning problems, such as active learning[14, 7], meta learning[53].

Link Prediction Models. Three primary approaches have been identified for link prediction models. Node embedding methods, as described by previous work [50, 30, 57, 23, 44, 45, 25], focus on

mapping each node to an embedding vector and leveraging these embeddings to predict connections. Another approach involves link prediction heuristics, as explored by [43, 15, 3, 77], which utilize crafted structural features and network topology to estimate the likelihood of connections between nodes in a network. The third category employs GNNs for predicting link existence; notable is the Graph Autoencoder (GAE) [36], which learns low-dimensional representations of graphstructured data through an unsupervised learning process. GAE utilizes the inner product of MPNN representations of target nodes to forecast links but might not capture pairwise relations between nodes effectively. More sophisticated GNN models that combine MPNN with additional structural features, such as those by [71, 70, 18], have demonstrated superior performance by integrating both node and structural attributes. One such combined architecture is SF-then-MPNN, as adopted by [71, 78]. In this approach, the input graph is first enriched with structural features (SF) and then processed by the MPNN to enhance its expressivity. However, since structural features change with each target link, the MPNN must be re-run for each link, reducing scalability. For instance, the SEAL model [71] first enhances node features by incorporating the shortest path distances and extracting k-hop subgraphs, then applies MPNN across these subgraphs to generate more comprehensive link representations. Another combined architecture is SF-and-MPNN. Models like Neo-GNN [70] and BUDDY [18] apply MPNN to the entire graph and concatenate features such as common neighbor counts to enhance representational fidelity. In addition, [61] has developed the Neural Common Neighbor with Completion (NCNC) which utilizes the MPNN-then-SF architecture to achieve higher expressivity and address the graph incompleteness.

Recently, representation learning on temporal graphs for link prediction has also been widely studied to exploit patterns in historical sequences, particularly with GNN-based methods [58, 69, 19, 64, 62, 55]. However, these approaches are still conventional supervised-learning-based methods that chronologically split the dataset into training and testing sets. Specifically, these methods train a GNN-based model on the temporal training data and then employ the trained model to predict links in the test data. In contrast, we formulate link predictions as sequential decision-making, where each link prediction is made sequentially. Node classification[16, 67, 66] is also a prominent direction in graph learning, but it is not the main focus of this paper.

3 Problem Definition

Let G 0 = ( V, E 0 ) be an undirected graph at initialization, where V is the set of n nodes, | V | = n , and E 0 ⊆ V × V represents the set of edges. E 0 can be an empty set in the cold-start setting or include some existing edges with a warm start. Each node v i ∈ V is associated with a context vector x 0 ,i ∈ R d . Then, we formulate link prediction as the problem of sequential decision-making under the framework of contextual bandits. Suppose the learner is required to finish a total of T link predictions. We adapt the above notation to all the evolving T graphs { G t = ( V, E t ) } T -1 t =0 and let [ T ] = { 1 , . . . , T } . In a round of link prediction t ∈ [ T ] , given G t -1 = ( V, E t -1 ) , the learner is presented with a serving node v t ∈ V and a set of k candidate nodes V t = { v t, 1 , . . . , v t,k } ⊆ V , where V t is associated with the corresponding k contexts X t = { x t, 1 , . . . , x t,k } and |V t | = k . In the scenario of social recommendation, v t can be considered as the user that the platform (learner) intends to recommend potential friends to, and the other candidate users will be represented by V t . V t can be set as the remaining nodes V t = V t /v t or formed by some pre-selection algorithm V t ⊂ V t .

The goal of the learner is to predict which node in V t will generate a link or edge with v t . Therefore, we can consider each node in V t as an arm, and aim to select the arm with the maximal reward or the arm with the maximal probability of generating an edge with v t . For simplicity, we define the reward of link prediction as the binary reward. Let v t, ˆ i ∈ V t be the node selected by the learner. Then, the corresponding reward is defined as r t, ˆ i = 1 if the link [ v t , v t, ˆ i ] is really generated; otherwise, r t, ˆ i = 0 . After observing the reward r t, ˆ i , we update E t -1 to obtain the new edge set E t , and thus new G t .

For any node v t,i ∈ V t , denote by D Y| x t,i the conditional distribution of the random reward r t,i with respect to x t,i , where Y = { 1 , 0 } . Then, inspired by the literature of contextual bandits, we define the following pseudo regret:

Algorithm 1 PRB (PageRank Bandits)

where i ∗ = arg max v t,i ∈V t P ( r t,i = 1 | x t,i ) , the tie is broken randomly, and ˆ i is the index of selected node. R T reflects the performance difference of the learned model from the Bayes-optimal predictor. The goal of the learner is to minimize R T .

4 Proposed Algorithms

Algorithm 1 describes the proposed algorithm PRB. It integrates contextual bandits and PageRank to combine the power of balancing exploitation and exploration with graph connectivity. The first step is to balance the exploitation and exploration in terms of the reward mapping concerning node contexts, and the second step is to propagate the exploitation and exploration score via graph connectivity.

To exploit the node contexts, we use a neural network to estimate rewards from the node contexts. Let f 1 ( · ; θ 1 ) be a neural network to learn the mapping from the node context to the reward. Denote the initialized parameter of f 1 by θ 1 0 . In round t , let θ 1 t -1 be parameter trained on the collected data of previous t -1 rounds including all selected nodes and the received rewards. Given the serving node v t , for any candidate node v t,i ∈ V t , f 1 ( x t,i ; θ 1 t -1 ) , i ∈ V t is the estimated reward by greedily exploiting the observed contexts, which we refer to as 'exploitation'. Suppose ˆ i is the index of selected nodes. To update θ 1 t -1 , we can conduct stochastic gradient descent to update θ 1 based on the collected training sample ( x t, ˆ i , r t, ˆ i ) with the squared loss function L ( x t, ˆ i , r t, ˆ i ; θ 1 t -1 ) = [ f ( x t, ˆ i ; θ 1 t -1 ) -r t, ˆ i ] 2 / 2 . Denote the updated parameters by θ 1 for the next round of link prediction.

t

In addition to exploiting the observed contexts, we employ another neural network to estimate the potential gain of each candidate node in terms of reward for exploration. This idea is inspired by [12]. Denote the exploration network by f 2 ( · ; θ 2 ) . f 2 is to learn the mapping from node contexts and the discriminative ability of f 1 to the potential gain. In round t ∈ [ T ] , given node context x t,i ∈ V t and its estimated reward f 1 ( x t,i ; θ 1 t -1 ) , the input of f 2 is the gradient of f 1 ( x t,i ; θ 1 t -1 ) with respect to θ 1 t -1 , denoted by ϕ ( x t,i ) , and f 2 ( ϕ ( x t,i ); θ 2 t -1 ) is the estimated potential gain. After the learner selects the node x t, ˆ i and observes the reward r t, ˆ i , the potential gain is defined as r t, ˆ i -f 1 ( x t,i ; θ 1 t -1 ) , which is used to train f 2 . Thus, after this interaction, we conduct the stochastic gradient descent to update θ 2 based on the collected sample ( ϕ ( x t, ˆ i ) , r t, ˆ i -f 1 ( x t,i ; θ 1 t -1 )) with the squared loss function

BinaryNodeClassification

LinkPredictionwithTwoSuperNodes

1

1

?

Classo

Super Node 0 (Class 0)

2

4

2

4

3

Class1

3

Super Node 1 (Class 1)

the updated parameters of f 2 for the next round of link prediction. The reasons for setting ϕ ( x t,i ) as the input of f 2 are as follows: (1) it incorporates the information of both x t, ˆ i and discriminative ability of f 1 ( · ; θ 1 t -1 ) ; (2) the statistical form of the confidence interval for reward estimation can be regarded as the mapping function from ϕ ( x t,i ) to the potential gain, and f 2 is to learn the unknown mapping [12].

The previous steps demonstrate the exploitation and exploration of node contexts to facilitate decisionmaking in link prediction. Since graph connectivity is also crucial, we next introduce our method of integrating the bandit principle with PageRank to enable collaborative exploitation and exploration. PageRank calculates the stationary distribution of the random walker starting from some node, iteratively moving to a random neighbor with probability α (damping factor) or returning to its original position with probability 1 -α . Let v t be the stationary distribution vector calculated based on the graph G t . Then, v t satisfies:

where P t ∈ E n × n is the transition matrix built on G t -1 and h t is typically regarded as a position vector to mark the starting node. P t is computed as D -1 t -1 A t -1 , where D t -1 ∈ R n × n is the degree matrix of G t -1 and A t -1 ∈ R n × n is the adjacency matrix of G t -1 .

Here we propose to use h t to include the starting exploitation and exploration scores of candidate nodes, defined as:

Therefore, v t is the vector for the final decision-making based on collaborative exploitation and exploration. Some research efforts have been devoted to accelerating the calculation of Eq.4.1 in the evolving graph, e.g., [42], which can be integrated into PRB (Line 9 in Algorithm 1) to boost its efficiency and scalability.

PRBfor Node Classification . Wealso extend PRB to solve the problem of node classification as illustrated in Figure 1. Consider a k -class classification problem. We add k super nodes { ˜ v 1 , ˜ v 2 , . . . , ˜ v k } to the graph, which represents k classes, respectively. Then, we transform the node classification problem into the link prediction problem, aiming to predict the link between the serving node and the k super nodes. To be specific, in round t ∈ [ T ] , the learner is presented with the serving node v t and the k candidate (super) nodes V t = { ˜ v 1 , ˜ v 2 , . . . , ˜ v k } associated with k corresponding contexts X t = { x t, 1 , x t, 2 , . . . , x t,k } . Recall x t is the context associated with v t . Then, we define the contexts of super nodes as x t, 1 = [ x ⊤ t , 0 , . . . , 0 ] ⊤ , x t, 2 = [ 0 , x ⊤ t , . . . , 0 ] ⊤ , . . . , x t,k = [ 0 , 0 , . . . , x t ] ⊤ , x t,i ∈ R kd , i ∈ [ k ] . This context definition is adopted from neural contextual bandits [12, 76]. Then, the learner is required to select one node from V t . Let ˜ v i t be the selected node and ˜ v i ∗ t be ground-truth node ( i ∗ t is the index of ground-truth class of node v t ). Then, after observing the reward r t,i t , one edge [ v t , ˜ v i t ] is added to the graph G t -1 , if v t belongs to the class i t , i.e., i t = i ∗ t and reward r t,i t = 1 . Otherwise, r t,i t = 0 and the edge [ v t , ˜ v i ∗ t ] is added to G t -1 . Then, we can naturally apply PRB to this problem. We detail our extended algorithm for node classification in Algorithm 2.

PRB Greedy. We also introduce a greedy version of PRB which integrates PageRank solely with contextual bandit exploitation, as outlined in Algorithm 3. We will compare each variant of algorithms in our experiment section.

5 Regret Analysis

In this section, we provide the theoretical analysis of PRB by bounding the regret defined in Eq.3.1. One important step is the definition of the reward function, as this problem is different from the standard bandit setting that focuses on the arm (node) contexts and does not take into account the graph connectivity. First, we define the following general function to represent the mapping from the node contexts to the reward. Given the serving node v t and an arm node v t,i ∈ V t associated with the context x t,i , the reward conditioned on v t and v t,i is assumed to be governed by the function:

where y is an unknown but bounded function that can be either linear or non-linear. Next, we provide the formulation of the final reward function. In round t ∈ [ T ] , let y t be the vector to represent the expected rewards of all candidate arms y t = [ y ( x t,i ) : v t,i ∈ V t ] . Given the graph G t -1 , its normalized adjacency matrix P t , and the damping factor α , inspired by PageRank, the optimizing problem is defined as: v ∗ t = arg min v α v ⊤ ( I -P t ) v +(1 -α ) ∥ v -y t ∥ 2 2 / 2 . Then, its optimal solution is

For any candidate node v t,i ∈ V t , we define its expected reward as E r t,i ∼D Y| x t,i [ r t,i ] = v ∗ t [ i ] . v ∗ t is a flexible reward function that reflects the mapping relation of both node contexts and graph connectivity. α is a hyper-parameter to trade-off between the leading role of graph connectivity and node contexts. When α = 0 , v ∗ t turns into the reward function in contextual bandits [76, 12]; when α = 1 , v ∗ t is the optimal solution solely for graph connectivity. Here, we assume α is a prior knowledge. Finally, the pseudo-regret is defined as

where i ∗ = arg max v t,i ∈V t v ∗ t [ i ] and ˆ i is the index of the selected node. The regret analysis is associated with the Neural Tangent Kernel (NTK) matrix as follows:

Definition 5.1 (NTK [34, 63]) . Let N denote the normal distribution. Given all data instances { x t } Tk t =1 , for i, j ∈ [ Tk ] , define

Then, the NTK matrix is defined as H = ( H L +Σ L ) / 2 .

Assumption 5.1. There exists λ 0 > 0 , such that H ⪰ λ 0 I .

The assumption 5.1 is generally made in the literature of neural bandits [76, 74, 20, 35, 12, 10, 65] to ensure the existence of a solution for NTK regression.

As the standard setting in contextual bandits, all node contexts are normalized to the unit length. Given x t,i ∈ R d with ∥ x t,i ∥ 2 = 1 , t ∈ [ T ] , i ∈ [ k ] , without loss of generality, we define a fully-connected network with depth L ≥ 2 and width m :

where σ is the ReLU activation function, W 1 ∈ R m × d , W l ∈ R m × m , for 2 ≤ l ≤ L -1 , W L ∈ R 1 × m , and θ = [ vec ( W 1 ) ⊤ , vec ( W 2 ) ⊤ , . . . , vec ( W L ) ⊤ ] ⊤ ∈ R p . Note that our analysis can also be readily generalized to other neural architectures such as CNNs and ResNet [5, 24]. We employ the following initialization [17] for θ : For l ∈ [ L -1] , each entry of W l is drawn from the normal distribution N (0 , 2 /m ) ; each entry of W L is drawn from the normal distribution N (0 , 1 /m ) . The network f 1 and f 2 follows the structure of f . Define y = [ y ( x t,i ) : t ∈ [ T ] , i ∈ [ k ]] . Finally, we provide the performance guarantee as stated in the following Theorem.

Theorem 5.1. Given the number of rounds T , for any α, δ ∈ (0 , 1) , suppose m ≥ ˜ Ω( poly ( T, L ) · k log(1 /δ )) , η 1 = η 2 = T 3 √ m and set ˜ r t,i = r t,i , t ∈ [ T ] , i ∈ [ k ] . Then, with probability at least 1 -δ over the initialization, Algorithm 1 achieves the following regret upper bound:

Theorem 5.1 provides a regret upper bound for PRB with the complexity of ˜ O ( ˜ d √ kT ) (see proofs in Appendix E). Instead, the graph-based methods (e.g., [18, 61]) lack an upper bound in terms of their performance. Theorem 5.1 provides insightful results in terms of PRB's performance. First, PRB's regret can grow sub-linearly with respective to T . Second, PRB's performance is affected by the number of nodes k . This indicates the larger the graph is, the more difficult the link prediction problem is. Third, ˜ d and S in the regret upper bound reflect the complexity of the required neural function class to realize the underlying reward function v ∗ t , i.e., the difficulty of learning v ∗ t . ˜ d is the effective dimension, which measures the actual underlying dimension in the RKHS space spanned by NTK. S is to provide an upper bound on the optimal parameters in the context of NTK regression. Both ˜ d and S are two complexity terms that commonly exist in the literature of neural contextual bandits[76, 74]. In the general case when 1 > α > 0 , learning v ∗ t proportionally turns into a bandit optimization problem and the upper bound provided in Theorem 5.1 matches the SOTA results in neural bandits [76, 74]. In fact, the regret upper bound is closely related to the graph structure of G t . In the special case when α = 1 , learning v ∗ t turns into a simple convex optimization problem (Eq. (4.1)) and PRB can really find the optimal solution, which leads to zero regrets. When α = 0 , the problem turns into a complete bandit optimization problem with the same regret upper bound as Theorem 5.1.

6 Experiments

In this section, we begin by conducting a comprehensive evaluation of our proposed method, PRB, compared with both bandit-based and graph-based baselines across online and offline link prediction settings. Then, we analyze the computational costs associated with each experiment and present additional ablation studies related to PRB. In the implementation of PRB, we adapt the efficient PageRank algorithm [42] to solve Eq. (4.1).

6.1 Online Link Prediction

MovieLens

2500

AmazonFashion

Facebook

4500

GrQc

1750

2500

4000

1500

2000

2000

3500

1250

750

EE-Net

1500

1000

EEnet

1000

EEnet

2500

2000

1500

3000

EEnet

500

250

Neural Greedy

NeuralTS

PRB

NeuralUCB

500

Neural Greedy

NeuralTS

PRB

NeuralUCB

500

NeuralTS

PRB

Neural Greedy

NeuralUCB

1000

500

Neural Greedy

NeuralTS

PRB

NeuralUCB

0

0

2000

4000

Rounds

6000

8000

10000

0

0

1000

2000

Rounds

3000

4000

5000

0

0

2000

4000

Rounds

6000

8000

10000

0

0

2000

4000

Rounds

6000

8000

10000

In this sub-section, we evaluate PRB on the setting of online link prediction and node classification as described in Sec. 3, compared with bandit-based baselines.

Datasets and Setups. We use three categories of real-world datasets to compare PRB with banditbased baselines. The details and experiment settings are as follows.

(1) Recommendation datasets: Movielens [32] and Amazon Fashion [48] (Bipartite Graph). Given the user set U and item set I , let G 0 be the graph with no edges, G 0 = ( V = U + I, E 0 = ∅ ) . In round t ∈ [ T ] , we randomly select a user v t ∈ U , and then randomly pick 100 items (arms) from I , including v t 's 10 purchased items, forming V t . PRB runs based on G t -1 and selects an arm (node) v t, ˆ i ∈ V t . If the selected arm v t, ˆ i is the purchased item by u t , the regret is 0 (or reward is 1 ) and we add the edge [ v t , v t, ˆ i ] to G t -1 , to form the new graph G t ; otherwise, the regret is 1 (or reward is 0 ) and G t = G t -1 .

(2) Social network datasets: Facebook [38] and GR-QC [37]. Given the user set V , we have G 0 = ( V, E 0 = ∅ ) . In a round t ∈ [ T ] , we randomly select a source node v t that can be thought of as the serving user. Then, we randomly choose 100 nodes, including v t 's 10 connected nodes but their edges are removed, which form the arm pool V t associated with the context set X t . Then, PRB will select one arm v t, ˆ i ∈ V t . If v t and v t, ˆ i are connected in the original graph, the regret is 0 and add the edge [ v t , v t, ˆ i ] to G t -1 ; otherwise, the regret is 1 and G t = G t -1 .

(3) Node classification datasets: Cora, Citeseer, and Pubmed from the Planetoid citation networks [68]. Recall the problem setting described in Sec. 4. Consider a k -class node classification problem. Given a graph G ( V, E 0 = ∅ ) , we randomly select a node v t ∈ V to predict its belonging class, in a round t ∈ [ T ] . Then, PRB select one super node ˜ v i t . If v t belongs to class i t , the regret is 0 and add [ v t , ˜ v i t ] to G t -1 . Otherwise, the regret is 1 and G t = G t -1 .

Baselines. For bandit-based methods, we apply Neural Greedy [12] that leverages the greedy exploration strategy on the exploitation network, NeuralUCB [76] that uses the exploitation network to learn the reward function along with an UCB-based exploration strategy, NeuralTS [74] that adopts the exploitation network to learn the reward function along with the Thompson Sampling exploration strategy, and EE-net [12] that utilizes the exploitation-exploration network to learn the reward function. Following [76, 12], for all methods, we train each network every 50 rounds for the first 2000 rounds and then every 100 rounds for the remaining rounds. See Appendix A.1 for additional experimental setups.

Online Link Prediction . We use Figure 2 to depict the regret trajectories over 10,000 rounds, and Table 1 to detail the cumulative regret after 10,000 rounds for all methods, where the lower is better. Based on the regret comparison, PRB consistently outperforms all other baselines across all datasets. For example, the cumulative regret at 10,000 rounds for PRB on MovieLens is considerably lower than the best-performing baseline, EE-Net. Similarly, in the AmazonFashion dataset, PRB achieved the lowest regret, surpassing the strongest baseline EE-Net over 14%. This trend is consistent across the Facebook and GrQc datasets, where PRB maintains its lead with the lowest regrets respectively. The consistency in PRB's performance across various datasets suggests the importance of utilizing the graph structure formed by previous link predictions.

Online Node classification . Figure 3 and Table 2 show the regret comparison on online node classification. PRB consistently demonstrates the lowest cumulative regret by outperforming other

3000

3500

Citeseer

2000

Pubmed

2500

3000

1750

2500

1500

2000

1250

Regret

2000

ret

1000

1500

1000

EEnet

EEnet

750

EEnet

Neural Greedy

1000

Neural Greedy

500

Neural Greedy

500

NeuraITS

NeuralUCB

500

NeuralUCB

EEnet

250

NeuralUCB

NeuralTS

PRB

PRB

PRB

0

0

2000

4000

6000

8000

10000

0

0

2000

4000

6000

8000

10000

0

0

2000

4000

6000

8000

10000

Rounds

Rounds

Rounds

bandit methods at round 10,000, respectively. Overall, PRB decreases regrets by 3.0%, 1.2%, and 3.5% compared to one of the best baselines, NeuralTS. This experiment demonstrates that PRB is versatile enough for applications beyond online link prediction, extending to other real-world tasks such as online node classification. This highlights PRB's advantage of fusing contextual bandits with PageRank for collaborative exploitation and exploration.

6.2 Offline Link Prediction

In this subsection, we evaluate PRB in the setting of offline link prediction compared with graph-based baselines, where training and testing datasets are provided, following the same evaluation process of [18, 61]. Here, we train PRB on the training dataset using the same sequential optimization method Sec. 6.1. Then, we run the trained PRB on the testing dataset. Notice that PRB never sees the test data in the training process as other baselines.

Datasets . In this study, we use real-world link-prediction datasets to compare PRB with graph-based baselines. Specifically, we apply Cora, Citeseer, and Pubmed from Planetoid citation networks [68]; ogbl-collab, ogbl-ppa, and ogbl-ddi from Open Graph Benchmark [33]. (See dataset statistics in Appendix C.)

Setting: We strictly follow the experimental setup in [18] and use the Hits@k metric for evaluation. Please also refer to A.1 for additional setups.

Baselines . For graph-based methods, we choose traditional link-prediction heuristics including CN [15], RA [77], AA [3] and common GNNs including GCN [36] and SAGE [31]. Then, we employ SF-then-MPNN models, including SEAL [71] and NBFNet [78], as well as SF-and-MPNN models like Neo-GNN [70] and BUDDY[18]. Additionally, we also select the MPNN-then-SF model NCN [61] and NCNC [61]. The results of the baselines are sourced from Table 2 of [61].

Comparison with Graph-based Baselines. We present the experimental results in Table 3 for all methods. The results demonstrate that PRB consistently outperforms other baselines across all six datasets. Specifically, compared to the most recent method, NCNC, PRB achieves a minimum improvement of 0.68% on the Collab dataset, a maximum of 4.2%, and an average of 2.42% across

all datasets. Given that all baselines lack the perspective of exploration, the results demonstrate that fusing the exploitation and exploration in contextual bandits along with learning graph connectivity through PageRank does significantly enhance accuracy for link prediction.

6.3 Ablation and Sensitivity Studies

Table 4 presents the performance of different variants of PRB, including PRB-greedy that only use the exploitation network and PRB-(10%-G) that has the warm start with addition 10% edges in G 0 . The results show that exploration is crucial to the final performance and the additional graph knowledge can boost the performance.

Due to the space limit, we move all other experiment sections to Appendix A, including computational cost analysis on PRB and additional ablation & sensitivity studies.

7 Conclusion

This paper introduces a fusion algorithm for link prediction, which integrates the power of contextual bandits in balancing exploitation and exploration with propagation on graph structure by PageRank. We further provide the theoretical performance analysis for PRB, showing the regret of the proposed algorithm can grow sublinearly. We conduct extensive experiments in link prediction to evaluate PRB's effectiveness, compared with both bandit-based and graph-based baselines.

Acknowledgments and Disclosure of Funding

This work is supported by the National Science Foundation under Award No. IIS-2117902 and DARPA (HR001121C0165). The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , pages 2312-2320, 2011.

Marc Abeille and Alessandro Lazaric. Linear thompson sampling revisited. In Artificial Intelligence and Statistics , pages 176-184. PMLR, 2017.

Lada A Adamic and Eytan Adar. Friends and neighbors on the web. Social networks , 25(3):211230, 2003.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning , pages 127-135. PMLR, 2013.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning , pages 242-252. PMLR, 2019.

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems , pages 8141-8150, 2019.

Yikun Ban, Ishika Agarwal, Ziwei Wu, Yada Zhu, Kommy Weldemariam, Hanghang Tong, and Jingrui He. Neural active learning beyond bandits. arXiv preprint arXiv:2404.12522 , 2024.

Yikun Ban and Jingrui He. Generic outlier detection in multi-armed bandit. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 913-923, 2020.

Yikun Ban and Jingrui He. Local clustering in contextual multi-armed bandits. In Proceedings of the Web Conference 2021 , pages 2335-2346, 2021.

Yikun Ban, Jingrui He, and Curtiss B Cook. Multi-facet contextual bandits: A neural network perspective. In The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021 , pages 35-45, 2021.

Yikun Ban, Yunzhe Qi, Tianxin Wei, Lihui Liu, and Jingrui He. Meta clustering of neural bandits. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 95-106, 2024.

Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. EE-net: Exploitation-exploration neural networks in contextual bandits. In International Conference on Learning Representations , 2022.

Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. Neural exploitation and exploration of contextual bandits. arXiv preprint arXiv:2305.03784 , 2023.

Yikun Ban, Yuheng Zhang, Hanghang Tong, Arindam Banerjee, and Jingrui He. Improved algorithms for neural active learning. Advances in Neural Information Processing Systems , 35:27497-27509, 2022.

Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. science , 286(5439):509-512, 1999.

Smriti Bhagat, Graham Cormode, and S Muthukrishnan. Node classification in social networks. Social network data analytics , pages 115-148, 2011.

Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in Neural Information Processing Systems , 32:10836-10846, 2019.

Benjamin Paul Chamberlain, Sergey Shirobokov, Emanuele Rossi, Fabrizio Frasca, Thomas Markovich, Nils Yannick Hammerla, Michael M Bronstein, and Max Hansmire. Graph neural networks for link prediction with subgraph sketching. In The eleventh international conference on learning representations , 2022.

Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mehrdad Mahdavi. Do we really need complicated model architectures for temporal networks? arXiv preprint arXiv:2302.11636 , 2023.

Zhongxiang Dai, Yao Shu, Arun Verma, Flint Xiaofeng Fan, Bryan Kian Hsiang Low, and Patrick Jaillet. Federated neural bandit. arXiv preprint arXiv:2205.14309 , 2022.

Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback, 2008.

Rohan Deb, Yikun Ban, Shiliang Zuo, Jingrui He, and Arindam Banerjee. Contextual bandits with online neural regression. arXiv preprint arXiv:2312.07145 , 2023.

Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. Data augmentation for deep graph learning: A survey. ACM SIGKDD Explorations Newsletter , 24(2):61-77, 2022.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning , pages 1675-1685. PMLR, 2019.

Dongqi Fu, Yikun Ban, Hanghang Tong, Ross Maciejewski, and Jingrui He. Disco: Comprehensive and explainable disinformation detection. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 4848-4852, 2022.

Dongqi Fu, Liri Fang, Ross Maciejewski, Vetle I. Torvik, and Jingrui He. Meta-learned metrics over multi-evolution temporal graphs. In Aidong Zhang and Huzefa Rangwala, editors, KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022 , pages 367-377. ACM, 2022.

Dongqi Fu and Jingrui He. SDG: A simplified and dynamic graph neural network. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, editors, SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021 , pages 2273-2277. ACM, 2021.

Chongming Gao, Kexin Huang, Jiawei Chen, Yuan Zhang, Biao Li, Peng Jiang, Shiqi Wang, Zhong Zhang, and Xiangnan He. Alleviating matthew effect of offline reinforcement learning in interactive recommendation. arXiv preprint arXiv:2307.04571 , 2023.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages 855-864, 2016.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) , 5(4):1-19, 2015.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems , pages 8571-8580, 2018.

Yiling Jia, Weitong Zhang, Dongruo Zhou, Quanquan Gu, and Hongning Wang. Learning neural contextual bandits through perturbed rewards. In International Conference on Learning Representations , 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. Graph evolution: Densification and shrinking diameters. ACM transactions on Knowledge Discovery from Data (TKDD) , 1(1):2-es, 2007.

Jure Leskovec and Julian Mcauley. Learning to discover social circles in ego networks. Advances in neural information processing systems , 25, 2012.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. Collaborative filtering bandits. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval , pages 539-548, 2016.

Zihao Li, Yuyi Ao, and Jingrui He. Sphere: Expressive and interpretable knowledge graph embedding for set retrieval. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang, editors, Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024 , pages 2629-2634. ACM, 2024.

Zihao Li, Dongqi Fu, and Jingrui He. Everything evolves in personalized pagerank. In Proceedings of the ACM Web Conference 2023 , pages 3342-3352, 2023.

David Liben-Nowell and Jon Kleinberg. The link prediction problem for social networks. In Proceedings of the twelfth international conference on Information and knowledge management , pages 556-559, 2003.

Xiao Lin, Jian Kang, Weilin Cong, and Hanghang Tong. Bemap: Balanced message passing for fair graph neural network. In Learning on Graphs Conference , pages 37-1. PMLR, 2024.

Xiao Lin, Zhining Liu, Dongqi Fu, Ruizhong Qiu, and Hanghang Tong. Backtime: Backdoor attacks on multivariate time series forecasting. arXiv preprint arXiv:2410.02195 , 2024.

Linyuan Lü and Tao Zhou. Link prediction in complex networks: A survey. Physica A: statistical mechanics and its applications , 390(6):1150-1170, 2011.

Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. arXiv preprint arXiv:1705.07347 , 2017.

Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantlylabeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages 188-197, 2019.

Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE , 104(1):11-33, 2015.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 701-710, 2014.

Yunzhe Qi, Yikun Ban, and Jingrui He. Neural bandit with arm group graph. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1379-1389, 2022.

Yunzhe Qi, Yikun Ban, and Jingrui He. Graph neural bandits. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1920-1931, 2023.

Yunzhe Qi, Yikun Ban, Tianxin Wei, Jiaru Zou, Huaxiu Yao, and Jingrui He. Meta-learning with neural bandit scheduler. Advances in Neural Information Processing Systems , 36, 2024.

Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arXiv:1802.09127 , 2018.

Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637 , 2020.

Yuan Sui, Jiaru Zou, Mengyu Zhou, Xinyi He, Lun Du, Shi Han, and Dongmei Zhang. Tap4llm: Table provider on sampling, augmenting, and packing semi-structured data for large language model reasoning. arXiv preprint arXiv:2312.09039 , 2023.

Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Largescale information network embedding. In Proceedings of the 24th international conference on world wide web , pages 1067-1077, 2015.

Yuxing Tian, Yiyan Qi, and Fan Guo. Freedyg: Frequency enhanced continuous-time dynamic graph model for link prediction. In The Twelfth International Conference on Learning Representations , 2023.

Hanghang Tong, Christos Faloutsos, and Jia-Yu Pan. Fast random walk with restart and its applications. In Sixth international conference on data mining (ICDM'06) , pages 613-622. IEEE, 2006.

Michal Valko, Nathaniel Korda, Rémi Munos, Ilias Flaounas, and Nelo Cristianini. Finite-time analysis of kernelised contextual bandits. arXiv preprint arXiv:1309.6869 , 2013.

Xiyuan Wang, Haotong Yang, and Muhan Zhang. Neural common neighbor with completion for link prediction. In The Twelfth International Conference on Learning Representations , 2023.

Yanbang Wang, Yen-Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. arXiv preprint arXiv:2101.05974 , 2021.

Zhilei Wang, Pranjal Awasthi, Christoph Dann, Ayush Sekhari, and Claudio Gentile. Neural active learning with performance guarantees. Advances in Neural Information Processing Systems , 34, 2021.

Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Inductive representation learning on temporal graphs. arXiv preprint arXiv:2002.07962 , 2020.

Pan Xu, Zheng Wen, Handong Zhao, and Quanquan Gu. Neural contextual bandits with deep representation and shallow exploration. arXiv preprint arXiv:2012.01780 , 2020.

Zhe Xu, Yuzhong Chen, Qinghai Zhou, Yuhang Wu, Menghai Pan, Hao Yang, and Hanghang Tong. Node classification beyond homophily: Towards a general solution. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2862-2873, 2023.

Zhe Xu, Boxin Du, and Hanghang Tong. Graph sanitation with application to node classification. In Proceedings of the ACM Web Conference 2022 , pages 1136-1147, 2022.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Le Yu, Leilei Sun, Bowen Du, and Weifeng Lv. Towards better dynamic graph learning: New architecture and unified library. Advances in Neural Information Processing Systems , 36:67686-67700, 2023.

Seongjun Yun, Seoyoon Kim, Junhyun Lee, Jaewoo Kang, and Hyunwoo J Kim. Neo-gnns: Neighborhood overlap-aware graph neural networks for link prediction. Advances in Neural Information Processing Systems , 34:13683-13694, 2021.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems , 31, 2018.

Muhan Zhang and Yixin Chen. Inductive matrix completion based on graph neural networks. arXiv preprint arXiv:1904.12058 , 2019.

Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems , 34:9061-9073, 2021.

Weitong Zhang, Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural thompson sampling. In International Conference on Learning Representations , 2021.

Lecheng Zheng, Baoyu Jing, Zihao Li, Hanghang Tong, and Jingrui He. Heterogeneous contrastive learning for foundation models and beyond. In Ricardo Baeza-Yates and Francesco Bonchi, editors, Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024 , pages 6666-6676. ACM, 2024.

Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning , pages 11492-11502. PMLR, 2020.

Tao Zhou, Linyuan Lü, and Yi-Cheng Zhang. Predicting missing links via local information. The European Physical Journal B , 71:623-630, 2009.

Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. Advances in Neural Information Processing Systems , 34:29476-29490, 2021.

Jiaru Zou, Mengyu Zhou, Tao Li, Shi Han, and Dongmei Zhang. Promptintern: Saving inference costs by internalizing recurrent prompt during large language model fine-tuning. arXiv preprint arXiv:2407.02211 , 2024.

A Additional Experiments

A.1 Experiment Setups

Online Link Prediction Setups . For all bandit-based methods including PRB, for fair comparison, the exploitation network f 1 is built by a 2-layer fully connected network with 100-width. For the exploration network of EE-Net and PRB, we use a 2-layer fully connected network with 100-width as well. For NeuralUCB and NeuralTS, following the setting of [76, 74], we use the exploitation network f 1 and conduct the grid search for the exploration parameter ν over { 0 . 001 , 0 . 01 , 0 . 1 , 1 } and for the regularization parameter λ over { 0 . 01 , 0 . 1 , 1 } . For the neural bandits NeuralUCB/TS, following their setting, as they have expensive computation costs to store and compute the whole gradient matrix, we use a diagonal matrix to make an approximation. For all grid-searched parameters, we choose the best of them for comparison and report the average results of 10 runs for all methods. For all bandit-based methods, we use SGD as the optimizer for the exploitation network f 1 . Additionally, for EE-Net and PRB, we use the Adam optimizer for the exploration network f 2 . For all neural networks, we conduct the grid search for learning rate over { 0 . 01 , 0 . 001 , 0 . 0005 , 0 . 0001 } . For PRB, we strictly follow the settings in [42] to implement the PageRank component. Specifically, we set the parameter α = 0 . 85 after grid search over { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 85 , 0 . 9 } , and the terminated accuracy ϵ = 10 -6 . For each dataset, we first shuffle the data and then run each network for 10,000 rounds ( t = 10 , 000 ). We train each network every 50 rounds when t < 2000 and every 100 rounds when 2000 < t < 10 , 000 .

Offline Link Prediction Setups . For the graph-based methods, we strictly follow the experimental and hyperparameters settings in [61, 18] to reproduce the experimental results. Offline link prediction task requires graph links to play dual roles as both supervision labels and message passing links. For all datasets, the message-passing links at training time are equal to the supervision links, while at test and validation time, disjoint sets of links are held out for supervision that are never seen during training. All hyperparameters are tuned using Weights and Biases random search, exploring the search space of hidden dimensions from 64 to 512, dropout from 0 to 1, layers from 1 to 3, weight decay from 0 to 0.001, and learning rates from 0.0001 to 0.01. Hyperparameters yielding the highest validation accuracy are selected, and results are reported on a single-use test set. For PRB, we use setups similar to those in the online setting. We utilize the exploitation network f 1 and exploration network f 2 both with 500-width. We set the training epoch to 100 and evaluate the model performance on validation and test datasets. We utilize the Adam optimizer for all baseline models. For PRB implementation, We utilize the SGD optimizer for f 1 and the Adam optimizer for f 2 .

A.2 Computational Cost Analysis

We conduct all of our experiments on an Nvidia 3060 GPU with an x64-based processor.

Time and space complexity . Let n be the number of nodes, t be the index of the current round of link prediction, k be the number of target candidate nodes, d be the number of context dimensions, and p be neural network width. For the online setting, let m t be the number of edges at round t . In the setting of online link prediction, the time complexity of PRB is O ( kdp + m t k ) , where the first term is the cost of calculating the exploitation-exploration score for each candidate node and the second term is the cost of running PageRank, following [42]. And, the space complexity is O ( n + m t ) to store node weights and edges. For the offline setting, let m be the number of edges in the testing dataset. Let F be the number target links to predict. Then, the inference time complexity of PRB for F links is O ( ndp ) + ˜ O ( mF ) . The first term is the cost of calculating the exploitation-exploration score for each node. The second term is the cost of PageRank [42]. The comparison with existing methods is listed in the following table:

In Figure 5, we analyze the running time of the internal components of PRB and PRB-Greedy (Algorithm 3). The comparison of the internal components reveals that the Random Walk phase accounts for 10% (PRB) and 6.3% (PRB-Greedy) on average of the total running time across seven datasets. Previous results also demonstrate that PRB significantly outperforms EE-Net which solely relies on the Exploitation-Exploration framework, by dedicating a small additional portion of time to the Random Walk component.

By recording the total training time of 10,000 rounds, we also compare PRB with other bandit-based baselines in Figure 4. Across all datasets, NeuralTS achieves the minimum average running time at 10.9 minutes, while PRB has the maximum at 17.5 minutes. Additionally, given that the Random

■Neural UCB

NeuralTS

Neural Greedy

EE-net

PRB-Greedy

PRB

30

25

20

/min

time/r

15

10

5

O

MovieLens

Facebook

GR-QC

AmazonFashion

Citeseer

Pubmed

Exploitation-Exploration

RandomWalk

Exploitation-Exploration

RandomWalk

GrQc

Amazon

Citeseer

Pubmed

GrQc

Amazon

Citeseer

Pubmed

Fashion

Fashion

Walk component takes only a minimal portion of our algorithm's running time, the average running times are relatively close between PRB-Greedy (15.4 minutes) and Neural Greedy (14.8 minutes), and between PRB (17.5 minutes) and EE-net (16.7 minutes). The comparative analysis reveals that while PRB incurs a relatively extended running time, it remains competitive with established baselines and demonstrates a significant enhancement in performance. This observation underscores the efficacy of PRB and supports its potential utility in practical applications despite its temporal demands.

Table 5 reports the inference time (one round in seconds) of bandit-based methods on three datasets for online link prediction. Although PRB takes a slightly longer time, it remains in the same order of magnitude as the other baselines. We adopt the approximated methods from [42] for the PageRank component to significantly reduce computation costs while ensuring good empirical performance.

Table 6 reports the inference time (one epoch of testing in seconds) of graph-based methods on three datasets for offline link prediction. PRB is faster than SEAL and shows competitive inference time as compared to other baselines.

2000

MovieLens

2000

AmazonFashion

1750

1750

1500

1500

1250

1250

750

750

500

500

PRB-Greedy

PRB-Greedy

250

PRB

250

PRB

PRB-Prior (10%-G)

PRB-Prior (10%-G)

0

0

2000

4000

6000

8000

10000

0

1000

2000

3000

4000

5000

Rounds

Rounds

2500

Facebook

4000

GrQc

3500

2000

3000

1500

2500

Regret

1000

1500

1000

500

PRB-Greedy

PRB-Greedy

PRB

500

PRB

PRB-Prior (10%-G)

PRB-Prior (10%-G)

0

0

2000

4000

6000

8000

10000

0

0

2000

4000

6000

8000

10000

Rounds

Rounds

2500

2500

Citeseer

2000

2000

1500

1500

Regret

Regret

1000

1000

500

PRB-Greedy

500

PPB1

PRB

PRB

PRB-Prior (10%-G)

PRB-Prior (10%-G)

0

0

2000

4000

0009

8000

10000

0

0

2000

4000

6000

8000

10000

Rounds

Rounds

2000

Pubmed

1750

1500

1250

750

500

PRB-Greedy

250

PRB

PRB-Prior (10%-G)

0

0

2000

4000

6000

8000

10000

Rounds

3000

MovieLens

2000

AmazonFashion

1750

2500

1500

2000

1250

750

1000

500

500

EE-Net

EvePPR

250

EEnet

EvePPR

PRB

PRB

0

0

2000

4000

6000

8000

10000

0

1000

2000

3000

4000

5000

Rounds

Rounds

2500

Facebook

4000

GrQc

3500

2000

3000

1500

2500

Regret

1000

1500

1000

500

EEnet

EEnet

EvePPR

500

EvePPR

PRB

PRB

0

0

2000

4000

6000

8000

10000

0

0

2000

4000

6000

8000

10000

Rounds

Rounds

2500

2500

Citeseer

2000

2000

1500

1500

Regret

Regret

1000

1000

500

EEnet

500

EEnet

EvePPR

EvePPR

PRB

PRB

0

2000

4000

6000

8000

10000

0

0

2000

4000

6000

8000

10000

Rounds

Rounds

2000

Pubmed

1750

1500

1250

750

500

-EEnet

250

EvePPR

PRB

0

0

2000

4000

6000

8000

10000

Rounds

A.3 Additional Ablation and Sensitivity Studies

PRB variants. To extensively evaluate PRB in our experiments, we provide the following variants. PRB is the direct implementation of Algorithm 1. The initial graph G 0 only contains all nodes without any edges. PRB-Greedy is the greedy version of Algorithm 1 by removing the exploration network, as specified in Algorithm 3. PRB-Prior (10%-G) is Algorithm 1 with prior knowledge by revealing 10% of training edges on the initial graph. We apply PRB-Prior in our experiments to demonstrate how extra prior knowledge about the graph improves PRB's decision-making process.

Figure 6 and Table 4 highlights the regret comparison of three PRB variants: PRB, PRB-Greedy, and PRB-Prior. For both online link prediction and node classification, PRB surpasses PRB-Greedy by an average of 5.8%, highlighting the robustness of the exploration network embedded within PRB. Additionally, in online link prediction, the PRB-Prior (10%-G) variant consistently outperforms its counterparts across a majority of datasets. This is particularly evident in the MovieLens and AmazonFashion datasets, where it achieves notably low cumulative regrets of 1521 and 1408. Same in online node classification, PRB-Prior (10%-G) demonstrates exceptional performance on two out of three datasets, recording cumulative regrets of 1804 in Cora and 2158 in Citeseer. These results emphasize the benefits of incorporating prior knowledge within PRB to enhance predictive accuracy.

Effectiveness of Bandits and PageRank. In Figure 7, we compare the performance of PRB with that of EvePPR [42] and EE-Net [12], which represent methodologies based on PageRank and contextual bandits respectively. On one hand, PRB significantly outperforms EvePPR by integrating the exploitation and exploration strategy, which enhances PageRank's decision-making capabilities. On the other hand, PRB surpasses EE-net by leveraging a more comprehensive understanding of the input graph's structure and connectivity through enhanced PageRank. Overall, PRB consistently achieves lower regrets compared to both EvePPR and EE-Net, demonstrating the effectiveness of combining the exploitation-exploration with PageRank.

B Limitations

In this paper, we propose the PRB algorithm that integrates the exploitation-exploration of contextual bandits with PageRank. We do not investigate other integration methods, such as combining such exploitation-exploration with other Random Walk algorithms or GNNs. We also evaluate PRB on online link prediction and node classification. Several other real-world tasks, such as Subgraph Matching and Node Clustering, remain unexplored. Our future research will extend PRB to these and additional related tasks [79, 56] to assess its broader implications.

C Graph Dataset Statistics

The statistics of each dataset are shown in Table 7. Random splits use 70%,10%, and 20% edges for training, validation, and test set respectively.

D Variant Algorithms

E Proof of Theorem 5.1

E.1 Preliminaries

Following neural function definitions and Lemmas of [13], given an instance x , we define the outputs of hidden layers of the neural network (Eq. (5.4)):

Then, we define the binary diagonal matrix functioning as ReLU:

✶ ✶ Accordingly, the neural network (Eq. (5.4)) is represented by

and

Here, given a constant R > 0 , we define the following function class:

Let L t represent the squared loss function in round t . We use x 1 , x 2 , . . . , x Tk represent all the context vectors presented in T rounds. Then, we define the following instance-dependent complexity term:

Then, we have the following auxiliary lemmas.

Lemma E.1. Suppose m,η 1 , η 2 satisfy the conditions in Theorem 5.1. With probability at least 1 -O ( TkL ) · exp( -Ω( mω 2 / 3 L )) over the random initialization, for all t ∈ [ T ] , i ∈ [ k ] , θ satisfying ∥ θ -θ 0 ∥ 2 ≤ ω with ω ≤ O ( L -9 / 2 [log m ] -3 ) , it holds uniformly that

Lemma E.2. Suppose m,η 1 , η 2 satisfy the conditions in Theorem 5.1. With probability at least 1 -O ( TkL ) · exp( -Ω( mω 2 / 3 L )) , for all t ∈ [ T ] , i ∈ [ k ] , θ, θ ′ (or Θ , Θ ′ ) satisfying ∥ θ -θ 0 ∥ 2 , ∥ θ ′ -θ 0 ∥ 2 ≤ ω with ω ≤ O ( L -9 / 2 [log m ] -3 ) , it holds uniformly that

Lemma E.3. Suppose m,η 1 , η 2 satisfy the conditions in Theorem 5.1. With probability at least 1 -O ( TkL ) · exp( -Ω( mω 2 / 3 L )) , for all t ∈ [ T ] , i ∈ [ k ] , θ, θ ′ satisfying ∥ θ -θ 0 ∥ 2 , ∥ θ ′ -θ 0 ∥ 2 ≤ ω with ω ≤ O ( L -9 / 2 [log m ] -3 ) , it holds uniformly that

Lemma E.4 (Almost Convexity) . Let L t ( θ ) = ( f ( x t ; θ ) -r t ) 2 / 2 . Suppose m,η 1 , η 2 satisfy the conditions in Theorem 5.1. With probability at least 1 - O ( TkL 2 ) exp[ -Ω( mω 2 / 3 L )] over randomness of θ 1 , for all t ∈ [ T ] , and θ, θ ′ satisfying ∥ θ -θ 0 ∥ 2 ≤ ω and ∥ θ ′ -θ 0 ∥ 2 ≤ ω with ω ≤ O ( L -6 [log m ] -3 / 2 ) , it holds uniformly that

where ϵ = O ( ω 4 / 3 L 3 √ log m ))

Lemma E.5 (User Trajectory Ball) . Suppose m,η 1 , η 2 satisfy the conditions in Theorem 5.1. With probability at least 1 -O ( TkL 2 ) exp[ -Ω( mω 2 / 3 L )] over randomness of θ 0 , for any R > 0 , it holds uniformly that

Lemma E.6 (Instance-dependent Loss Bound) . Let L t ( θ ) = ( f ( x t ; θ ) -r t ) 2 / 2 . Suppose m,η 1 , η 2 satisfy the conditions in Theorem 5.1. With probability at least 1 -O ( TkL 2 ) exp[ -Ω( mω 2 / 3 L )] over randomness of θ 1 , given any R > 0 it holds that

where θ ∗ = arg inf θ ∈ B ( θ 0 ,R ) ∑ T t L t ( θ ) .

E.2 Regret analysis

Lemma E.7. Suppose m,η 1 , η 2 satisfies the conditions in Theorem 5.1. In round t ∈ [ T ] , let ˆ i be the index selected by the algorithm. Then, For any δ ∈ (0 , 1) , R > 0 , with probability at least 1 -δ , for t ∈ [ T ] , it holds uniformly

where H t = { x τ, ˆ i , r τ, ˆ i } t τ =1 represents of historical data selected by π τ and expectation is taken over the reward.

Proof. First, according to Lemma E.5, θ 2 0 , . . . , θ 2 T -1 all are in B ( θ 0 , R/m 1 / 4 ) . Then, according to Lemma E.1, for any x ∈ R d , ∥ x ∥ 2 = 1 , it holds uniformly | f 1 ( x t, ˆ i ; θ 1 t ) + f 2 ( ϕ ( x t, ˆ i ); θ 2 t ) -r t, ˆ i | ≤ O (1) .

Then, for any τ ∈ [ t ] , define

Then, we have

where F τ -1 denotes the σ -algebra generated by the history H τ -1 .

Therefore, the sequence { V τ } t τ =1 is the martingale difference sequence. Applying the HoeffdingAzuma inequality, with probability at least 1 -δ , we have

As I 1 is equal to 0 , we have

For I 3 , based on Lemma E.6, for any θ ′ satisfying ∥ θ ′ -θ 2 0 ∥ 2 ≤ R/m 1 / 4 , with probability at least 1 -δ , we have

where ( a ) is based on the definition of instance-dependent complexity term. Combining the above inequalities together, with probability at least 1 -δ , we have

The proof is completed.

Corollary E.1. Suppose m,η 1 , η 2 satisfy the conditions in Theorem 5.1. For any t ∈ [ T ] , let i ∗ be the index selected by some fixed policy and r t,i ∗ is the corresponding reward, and denote the policy by π ∗ . Let θ 1 , ∗ t -1 , θ 2 , ∗ t -1 be the intermediate parameters trained by Algorithm 1 using the data select by π ∗ . Then, with probability at least (1 -δ ) over the random of the initialization, for any δ ∈ (0 , 1) , R > 0 , it holds that

where H ∗ τ -1 = { x τ,i ∗ , r τ,i ∗ } τ -1 τ ′ =1 represents the historical data produced by π ∗ and the expectation is taken over the reward.

Define g ( x t ; θ ) ∇ θ = f ( x t ; θ ) for brevity.

Lemma E.8. Suppose m satisfies the conditions in Theorem 5.1. With probability at least 1 -δ over the initialization, there exists θ ′ ∈ B ( θ 0 , ˜ Ω( T 3 / 2 )) , such that

Proof.

where ( a ) is based on Lemma E.9 and ( b ) is an application of Lemma 11 in [1] and Lemma E.12, and O (1) is induced by the choice of m . By ignoring O (1) , The proof is completed.

Definition E.1. Given the context vectors { x i } T i =1 and the rewards { r i } T i =1 , then we define the estimation ̂ θ t via ridge regression:

Lemma E.9. Suppose m satisfies the conditions in Theorem 5.1. With probability at least 1 -δ over the initialization, there exists θ ′ ∈ B ( θ 0 , ˜ Ω( T 3 / 2 )) for all t ∈ [ T ] , such that

Proof. Given a set of context vectors { x } T t =1 with the ground-truth function h and a fully-connected neural network f , we have

where θ ′ is the estimation of ridge regression from Definition E.1. Then, based on the Lemma E.10, there exists θ ∗ ∈ R P such that h ( x i ) = ⟨ g ( x i , θ 0 ) , θ ∗ ⟩ . Thus, we have

where the final inequality is based on the the Theorem 2 in [1], with probability at least 1 -δ , for any t ∈ [ T ] .

Second, we need to bound

To bound the above inequality, we first bound

where we initialize f ( x t ; θ 0 ) = 0 and the inequality is derived by Lemma E.2 with ω = O ( t 3 / 2 ) m 1 / 4 . Next, we need to bound

Due to the Lemma E.12 and Lemma E.11, we have

Finally, putting everything together, we have

The proof is completed.

Definition E.2.

G (0) and r are formed by the selected contexts and observed rewards in T rounds, G 0 are formed by all the presented contexts.

Inspired by Lemma B.2 in [76] , with η = m -1 / 4 we define the auxiliary sequence following :

Lemma E.10. Suppose m satisfies the conditions in Theorem 5.1. With probability at least 1 -δ over the initialization, for any t ∈ [ T ] , i ∈ [ k ] , the result uniformly holds:

Proof. Based on Lemma E.13 with proper choice of ϵ , we have

Define h = [ h u 1 ( x 1 ) , . . . , h u T ( x Tk )] . Suppose the singular value decomposition of G 0 is PAQ ⊤ , P ∈ R p × Tk , A ∈ R Tk × Tk , Q ∈ R Tk × Tk , then, A ⪰ 0 . Define θ ∗ = θ 0 + PA -1 Q ⊤ h .

Then, we have which leads to

Therefore, the result holds:

Lemma E.11. There exist θ ′ ∈ B ( θ 0 , ˜ O ( T 3 / 2 L + √ T )) , such that, with probability at least 1 -δ , the results hold: √

Proof. The sequence of θ ( j ) is updates by using gradient descent on the loss function:

For any j > 0 , the results holds:

where the last inequality is held by Lemma E.1. Finally, given the j > 0 ,

For (2), by standard results of gradient descent on ridge regression, θ ( j ) , and the optimum is θ (0) + ̂ θ t . Therefore, we have

By setting λ = 1 and j = log(( T + O ( T 2 L )) -1 ) / log(1 -m -1 / 4 ) , we have ∥ θ ( j ) -θ 0 -̂ θ t ∥ 2 2 ≤ 1 1+ O ( TL ) . Replacing k and λ in (E.16) finishes the proof.

Lemma E.12. Suppose m satisfies the conditions in Theorem 5.1. With probability at least 1 -δ over the initialization, the result holds:

Proof. Based on the Lemma E.1, for any t ∈ [ T ] , ∥ g ( x t ; θ 0 ) ∥ 2 ≤ O ( √ L ) . Then, for the first item:

Next, we have

Then, we have

Accordingly, we have

The first inequality is because the concavity of log det ; The third inequality is due to ∥ ( I + H λ ) -1 ∥ F ≤ ∥ I -1 ∥ F ≤ √ T ; The last inequality is because of the choice the m , based on Lemma E.13; The last equality is because of the Definition of ˜ d . The proof is completed.

Lemma E.13. For any δ ∈ (0 , 1) , if m = Ω ( L 6 log( TkL/δ ) ( ϵ/Tk ) 4 ) , then with probability at least 1 -δ , the results hold:

Proof. This is an application of Lemma B.1 in [76] by properly setting ϵ

.

Lemma E.14 (Exactness of PageRank [42]) . When PageRank achieves the stationary distribution, v t = 1 -α I -α P t h t .

Finally, we provide the proof of Theorem 5.1.

Proof.

where (1) is by the choice of PRB and (2) is based on the exact solution of PageRank [42]. Let λ max be the maximal eigenvalue of P t . Because P t is a stochastic matrix, λ max = 1 . Then, we have

Then, based on Corollary E.1, Lemma E.8, and Lemma E.3, with shadow parameters, we have

To sum up, we have

The proof is completed.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have included discussion for our contributions in the Abstract and Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have included the discussion of the limitation in the Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please see theoretical analysis section and appendix for details.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please see appendix and our submitted source code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include the source code along with our submission.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have shown detailed experimental settings/details in the Experiment and Appendix section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have provided standard deviation as part of our experiment results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see the appendix where we mention our computational infrastructure and time of execution.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm this perform conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper does not perform societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not include any data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make the best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have credited correctly for the existing data and models we referred.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new datasets are introduced.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects are involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects are involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose PRB, a new method that blends exploration, exploitation from previous neural bandit literature into an architecture that effectively considers graph connectivity in order to boost the performance for both the node classification and link prediction tasks. The authors demonstrate the soundness of the new model with proofs and mathematical reasoning, which also distinguishes it from previous neural bandit research. Additionally, the authors verify the effectiveness of the model by measuring it's performance in online and offline link prediction along with node classification.\n\nSTRENGTHS:\n* Well-written paper that explains PRB's functionality and also highlights the difference between the problem solved by this paper and other neural bandit papers.\n* PRB empirically tested against SOTA baselines across online and offline link prediction and tested on node classification.\n\nWEAKNESSES:\n* Clarity of descriptions is sometimes more dense than necessary, this could be due to the page limitations. For example, Section 4 can be divided into subsections that distinguishes whether it is describing the exploration, exploitation, graph connectivity/bandit princple in PageRank. More examples are determined line-by-line in the questions section of this review.\n* Lack of inclusion of code for reproducing offline link prediction results on the OGBL Datasets in supplementary material. This is the largest contributing factor to the score provided in this review, further explanation about this concern is detailed in 'Concerns about PRB performance' within the Questions section of this review.\n\nQUESTIONS:\n* Line 135: Are there specific citations to contextual bandits literature that inspires the proposed pseudo-regret metric?\n* Line 179-180: This statement is certainly interesting, but how does it relate to PRB directly?\n\nConcerns about Clarity:\n------------------------------\n* Line 181-193: Do the super nodes allow PRB to extend to the node classification? From node classification, is PRB then updated to handle link prediction by considering the connections between serving nodes and super nodes? This seems to be the case, since the papers moves from the initial definition of PRB's neural bandit-style architecture onto PRB's applications within the link prediction and node classification tasks. It is difficult to tell without more context or definitions on the relation between super nodes and PRB. Additionally, this paragraph reads much like a mathematical proof, which is good to represent individual components within the system. However, a diagram on how PRB is applied to the node classification task and then how PRB is transformed for the link prediction task would provide clarity through a visual example.\n\n\nConcerns about PRB performance: \n----------------------------------------------\n* Given how neural bandit models are applied to online scenarios, I understand that the metric of choice for testing neural-bandit methods is regret (or pseudo-regret for PRB). Is it possible to test the accuracy of PRB for offline node classification? This is not necessarily a concern, but offline node classification results could provide insight into whether PRB's performance with pseudo-regret can translate to high-levels of accuracy.\n* As mentioned in Lines 242-243: the larger the input graph, the more difficult the link prediction task is for PRB. Does that mean that the more links which PRB is required to predict, the more components of the graph PRB is then required to exploit and explore? Is there another reason why neural bandit models have difficulty with larger graphs? Do these difficulties mean neural bandits suffer: when performing link prediction and node classification, with time and space complexity limiitations, or something else?\n* Considering Lines 242-243, along with PRB outperforming all other tested models, the lack of a script in the current supplementary materials to test how well PRB performs with the Hits@K metric on the much larger ogbl datasets is concerning. I kindly request the authors provide a script for replicating the results detailed in Table 3 for PRB and another model such as BUDDY, along with each model's recommended hyperparameters for the ogbl datasets.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-20"}

comments: SUMMARY:
This paper introduces PRB (PageRank Bandits), a novel algorithm for link prediction in graph learning that combines contextual bandits with PageRank to balance exploitation and exploration. Framing link prediction as a sequential decision-making process, the paper provides a new reward formulation and theoretical performance guarantees. PRB demonstrates superior performance over traditional methods in both online and offline evaluations.

STRENGTHS:
1. The integration of PageRank with contextual bandits is a compelling concept, and the motivation behind the proposed PRB algorithm is clear.
2. Regret analysis validates the PRB algorithm's efficacy, demonstrating that its cumulative regret approaches sublinear growth.
3. The PRB algorithm exhibits robust performance across real-world tasks, achieving impressive outcomes in both online and offline settings.
4. The organization of this paper is well-structured and straightforward, facilitating ease of comprehension.

WEAKNESSES:
1. The paper does not provide detailed descriptions of experimental settings, such as model hyperparameter configurations and optimizer choices.
2. The complexity analysis of the PRB algorithm, including both time and space complexity, is not thoroughly discussed.

QUESTIONS:
1. Please respose the above weaknesses first.
2. In practical experiments, how are the transition matrix and the parameter $\alpha$ chosen in the PRB algorithm?

LIMITATIONS:
The authors discussed the limitations in the Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-09

comments: SUMMARY:
This paper reformulates link prediction as a sequential decision-making process and propose a algorithm that combines contextual bandits with PageRank for collaborative exploitation and exploration. The experiments validate the effectiveness of the method.

STRENGTHS:
1. The problem is interesting and the paper is organized well.
2. The experiments show the effectiveness of the method.

WEAKNESSES:
1. In the introduction, some recent works use GNN models to investigate the graph embedding in temporal networks and the embedding also evolves over time and is applicable to various downstream tasks. However, the relevant works are missing.
2. For the node classification, how to pre-determine the number of clusters?
3. For the PageRank, whether the hyperparameter $\alpha$ need to be learned, and how to determine the best value?
4. Link prediction is well studied problem, why do we really need bandits to study this problem? The proposed reason is not very convincing.

QUESTIONS:
See above.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn light of the dynamic and evolving nature of real-world graphs, PageRank Bandits is proposed to make the prediction task consistently meet the context and adapt over time. Both experimental results and theoretical analysis are solid. But the paper organization is not so well.\n\nSTRENGTHS:\n--It is novel to combine contextual bandits with PageRank to find a better tradeoff between exploitation and exploration with graph connectivity.\n\n--Both contextual bandits and link prediction models are listed in related work. Moreover, both kinds of methods are compared as baselines in the experiments.\n\n--Comprehensive experiments are conducted to verify PageRank Bandits’ effectiveness and ablation study is also done to indicate that the model design is reasonable.\n\n--Theoretical analysis is also provided in the paper and the appendix to show its feasibility.\n\nWEAKNESSES:\n--The advantage of solving the link prediction task in a bandit setting is unclear. As mentioned in Paragraph 2 from Line 29, the dynamic and evolving nature of real-world graphs should be captured in the link prediction model. Experimental result analysis is short. For example, the failure of graph based baselines is owing to lack of exploration in Line 339. It is hard to understand this single sentence because that the graph neural network provides the capability of message passing.\n\n--Contextual bandits have been widely deployed in practice for online personalization and recommendation tasks.\n\n--Experimental setting is too simple to be self-contained. Training details are missing. \n\n--Font sizes of figures in the appendix are small.\n\n--The ablation study of bandits and PageRank should be included in the experiments to verify its main claim that the combination of contextual bandits and PageRank to achieve the balancing between exploitation and exploration.\n\n--The inference time should also be compared between the bandit methods and the graph based methods for link prediction in the experiments.\n\nQUESTIONS:\n--The inference time should also be compared between the bandit methods and the graph based methods for link prediction in the experiments.\n\nLIMITATIONS:\nLimitation should be provided.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}