# Fair Benchmark for Unsupervised Node Representation Learning
## 2 Fair and Comprehensive Evaluation Algorithm--FURL
### Task 2--Link Prediction Fine Tuned in the First Run  
Link prediction has also been widely used to evaluate embedding models. In this task, we evaluate the effectiveness of each NRL algorithm in preserving network patterns by  
Figure 2: Given a node representation learning method (e.g., Method A), FURL could fairly and comprehensively evaluate it based on its performance on three tasks and many real-world networks/datasets. The three tasks are classification fine tuned via a validation set, link prediction fine-tuned in the first run, and classification fine tuned via link prediction. In each task, the hyperparameters of Method A are fine-tuned based on a random search.  
applying the learned representations to recover unseen links. As illustrated in Figure 2, Task 2 consists of three steps. First, given a network, e.g., Net1, we randomly select \(B/2\) node pairs that are connected as positive samples, and \(B/2\) node pairs that are not connected as negative samples, where B denotes the total number of node pairs to be predicted. We remove the \(B/2\) edges of the \(B/2\) positive samples from the original network. Then, we apply the given NRL model, e.g., Method A, to embed the new network. We will employ the learned node representations to perform link prediction. Second, in the first run, we tune the hyperparameters of Method A. In particular, we mix the \(B/2\) positive samples and \(B/2\) negative samples and obtain a set with \(B\) node pairs. For each pair in the \(B\) node pairs, we denote its two nodes as \(i\) and \(j\). We compute the inner product of the embedding representations of nodes \(i\) and \(j\), and employ the inner product to indicate the probability of having a link between nodes \(i\) and \(j\). After obtaining the inner products of all \(B\) pairs, we rank them and compute the average precision (AP) and area under the receiver operating characteristic curve (ROC AUC), indicating the performance of link prediction. Based on this link-prediction performance, we select the best hyperparameters for the embedding model, i.e., Method A. Third, we employ the selected best hyperparameters to conduct another ten runs. In each run, we randomly select \(B/2\) positive samples and \(B/2\) negative samples from the original network. We remove the \(B/2\) edges of positive samples from the original network, and obtain a new network. We apply Method A with the selected best hyperparameters to embed the new network. We calculate the inner products of all \(B\) pairs, and we compute the AP and ROC AUC. The average of the results of the ten runs is employed as the final performance of Method A in Task 2.  
In summary, we fine tune Method A in the first run and obtain the best hyperparameters. We employ the selected hyperparameters to perform another ten runs. In each run, a different set of \(B\) samples are utilized to conduct the evaluation, that is, a different network (after removing \(B/2\) edges) is used to conduct embedding. Both the hyperparameter tuning and final evaluation are conducted in an unsupervised manner.