Label-free Node Classification on Graphs with Large Language Models (LLMs)
Authors: Zhikai Chen
Haitao Mao
Hongzhi Wen
Haoyu Han
Wei Jin
Haiyang Zhang
Hui Liu
Jiliang Tang
Venue: ICLR 2024 poster (2024)
Keywords: Node classification
Graph Neural Networks
Large Language Models
Primary area: learning on graphs and other geometries & topologies

Abstract: In recent years, there have been remarkable advancements in node classification achieved by Graph Neural Networks (GNNs). However, they necessitate abundant high-quality labels to ensure promising performance. In contrast, Large Language Models (LLMs) exhibit impressive zero-shot proficiency on text-attributed graphs. Yet, they face challenges in efficiently processing structural data and suffer from high inference costs. In light of these observations, this work introduces a label-free node classification on graphs with LLMs pipeline, LLM-GNN. It amalgamates the strengths of both GNNs and LLMs while mitigating their limitations. Specifically, LLMs are leveraged to annotate a small portion of nodes and then GNNs are trained on LLMs' annotations to make predictions for the remaining large portion of nodes. The implementation of LLM-GNN faces a unique challenge: how can we actively select nodes for LLMs to annotate and consequently enhance the GNN training? How can we leverage LLMs to obtain annotations of high quality, representativeness, and diversity, thereby enhancing GNN performance with less cost?
To tackle this challenge, we develop an annotation quality heuristic and leverage the confidence scores derived from LLMs to advanced node selection. Comprehensive experimental results validate the effectiveness of LLM-GNN. In particular, LLM-GNN can achieve an accuracy of 74.9\% on a vast-scale dataset \products with a cost less than 1 dollar.

Published as a conference paper at ICLR 2024
LABEL-FREE NODE CLASSIFICATION ON GRAPHS WITH LARGE LANGUAGE MODELS (LLMS)
Zhikai Chen', Haitao Mao', Hongzhi Wen', Haoyu Han', Wei Jin", Haiyang Zhang", Hui Jiliang Tang'
Michigan State University EEmory University 3 AAmazon.com {chenzh85, haitaoma, wenhongz, hanhaoy1, liuhui7 tangjili @ msu.edu, @eujiiiemmry,duu, hhaiz amazon.com
ABSTRACT
In recent years, there have been remarkable advancements in node classification achieved by Graph Neural Networks (GNNs). However, they necessitate abun- dant high-quality labels to ensure promising performance. In contrast, Large Lan- guage Models (LLLMs) exhibit impressive zero-shot proficiency on text-attributed graphs. Yet. they face challenges in efficiently processing structural data and suffer from high inference costs. light of these observations, this work in- troduces label-free node classification GNN. It amalgamates the strengths of both GNNs and LLMs while mitigating their limitations. Specifically, LLMs are leveraged annotate small portion of nodes and then GNNs are trained on LLMs annotations to make predictions for the remaining large portion of nodes. The implementation of LLM-GNN faces unique challenge: how can we actively select nodes for LLMs to anno- tate and consequently enhance the GNN training? How can we leverage LLMs to obtain annotations of high quality, representativeness. and diversity, thereby enhancing GNN performance with less cost? To tackle this challenge, we de- velop annotation quality heuristic and leverage the confidence scores derived from LLMs to advanced node selection. Comprehensive experimental results val- idate the effectiveness of LLM-GNN on text-attributed graphs from various do- mains. In particular, LLM-GNN can achieve an accuracy of 74.9% on a vast-scale dataset OGBN-PRODUCTS with cost less than 1 dollar. Our code is available from
graphs with LLMs pipeline, LLM-
com wwwpeooooootooo//oo/ooe/eo
INTRODUCTION
Graphs prevalent across multiple disciplines with diverse applications (Ma & Tang 1021). A graph is composed of nodes and edges, and nodes often with certain attributes, especially text at- tributes, representing properties of nodes. For example, in the OGRN-PRODUCTS dataset (Hu al.) 2020b), each node represents a product, and its corresponding textual description corresponds to the node's attribute. Node classification beled nodes based on part of labeled nodes, node attributes, and graph structures. In recent years, Graph Neural Networks (GNNs) have achieved superior performance node classification (Kipf & Welling, 2016 Hamilton et al., 2017 Veličković et al. 2017). Despite the effectiveness of GNNs, they always assume the ready availability of ground truth labels as a prerequire. Particularly, such assumption often neglects the pivotal challenge of procuring high-quality labels for graph- strpt-str data: (1) given the diverse and complex nature of graph- graph-structured data, human labeling inherently hard; (2) given the sheer scale of real-world graphs, such
critical task for graphs which aims assign labels unla-
(Hu et al. (020b) with millions of nodes, the process of annotating a significant portion of the nodes becomes both time-consuming and resource-intensive.
Compared to GNNs which require adequate high-quality labels, Large Language Models (LLMs) with massive knowledge have showcased impressive zero-shot and few-shot capabilities, especially for the node classification task on aext-bttributed graphs (TAGs) (Guo et al.) 2023] Chen et al.) 2023) He al. 2023a). Such evidence suggests that L.LMs achieve promising performance with-

---

Published as a conference paper at ICLR 2024
out the requirement for any labeled data. However, unlike GNNs, LLMs cannot naturally capture and understand informative graph structural patterns (Wang et al. 2023a). Moreover, LLMs can not be well-tuned since they can only utilize limited labels due to the limitation of input context length (Dong et al. 2022). Thus, though L.MS can achieve promising performance in zero-shot or few-shot scenarios, there may still be performance gap between L.MMs and GNNs trained with abundant labeled nodes (Chen et al.) 2023). Furthermore, the prediction cost of LLMs is much higher than that of GNNs, making it less scalable for large datasets such as OGBN-ARXIV and OGEN-PRODUCTS (Hu et al., 2020b).
In summary, we make two primary observations: (1) Given adequate annotations with high qual- ity, GNNs excel in utilizing graph structures to provide predictions both efficiently and effectively. Nonetheless, limitations can be found when adequate high-quality annotations are absent. (2) In contrast, LLMs can achieve satisfying performance without high-quality annotations while being costly. Considering these insights, becomes evident that GNNs and LLMs possess complemen- tary strengths. This leads us to an intriguing question: Can we harness the strengths of both while addressing their inherent weaknesses?
In this paper, we provide an affirmative answer to the above question by investigating the potential of harnessing the zero-shot learning capabilities of LLMs to alleviate the substantial training data demands of GNNs, scenario we refer as label-free node classification. Notably, unlike the common assumption that ground truth labels are always available, noisy labels can be found when annotations are generated from LLMs. We thus confront unique challenge: How can we ensure the high quality of the annotation without sacrifice diversity and representativeness? On one hand, are required consider the design of appropriate prompts enable LLMs to produce more accurate annotations. On the other hand, we need to strategically choose a set of training nodes that not only possess high-quality annotations but also exhibit informativeness and representativeness, as prior research has shown correlation between these attributes and the performance of the trained model (Huang et al. 2010).
To overcome these challenges, we propose a label-free node classification on graphs with LLMs pipeline, LLM-GNN. Different from traditional graph active node selection (Wu et al. 2019 Cai et al. 2017), LLM-GNN considers the node annotation difficulty by LLMs to actively select nodes. Then, it utilizes LLMS to generate confidence-awarc annotations and leverages the confidence score to further refine the quality of annotations post- fist-filtrrngg By seamlessly blending annotation quality with active selection, LLM-GNN achieves impressive results at a minimal cost, eliminating the necessity for ground truth labels. Our main contributions be summarized as follows:
1. We introduce a new label-free pipeline LLMM-NNN to leverage LLMs for annotation, providing training signals GNN for further prediction.
2. We adopt LLMS to generate annotations with calibrated confidence, and introduce difficulty- aware active selection with post filtering to get training nodes with a proper trade-off between annotation quality and traditional graph active selection criteria.
On the massive-scale OGBN-PRODUSE out the need for human annotations. This performance is comparable to manually annotating 400 randomly selected nodes, while the cost of the annotation process via I.LMs is under 1 dollar.
dataset, LLM-GNN can achieve 74.9% accuracy with-
2 PRELIMINARIES
In this section, we introduce text-attributed graphs and notation utilized in our study. We then review two primary pipelines on node classification. The first pipeline is the default node classification pipeline to evaluate the performance of GNNs (Kipf & Welling, 2016). while it totally ignores the data selection process. The second pipeline further emphasizes the node selection process, trying to identify the most informative nodes as training sets to maximize the model performance within a given budget.
Our study focuses on Text-Attributed Graph (IAG), represented as Gr (V.A.T,X). V {ui,..., , } is the set of n nodes paired with raw attributes T = {t1, t2...., .e....,tty ). Each text attributes can then be encoded as sentence embedding X {I1,
.n) with the help of Sen- tenceBERT (Reimers& & Gurevych 2019). The adjacency matrix A E {0, 1) represents graph connectivity where A (. = indicates an edge between nodes i and j. Although our study puts more emphasis on TAGs, it has the potential to be extended to more types of graphs through methods
like Liu et al. 2003) and /hhao et al. (2023).
2

---

Published as a conference paper at ICLR 2024
Traditional GNN-based node classification pipeline. assumes fixed training with ground truth labels yv for the training set Verain- The GNN is trained on those graph truth labels. The well-trained GNN predicts labels of the rest unlabeled nodes V Vivain in the test stage.
Traditional graph active learning-basee node classification. aims to select a group of nodes Vau (A, X) from the pool V so that the performance of GNN models trained those graphs with labels yva. can be maximized.
Limitations of the current pipelines. Both pipelines above assume they always obtain ground truth labels (Zhang et al., 2021c| WW et al., 2019) while overlooking the intricacies of the anno- tation process. Nonetheless, annotations can be both expensive and error-prone in practice, for seemingly straightforward tasks (Wei et al. (022). For example, the accuracy of human an- notations for the CIFAR-10 dataset approximately 82%, which only involves the categorization of daily objects. Annotation on graphs meets its unique challenge. Recent evidence (Zhu et al.) 2021a) shows that the human annotation on graphs easily biased, focusing nodes sharing some characteristics within small subgraph. Moreover, it can be even harder when taking graph active learning into consideration, the improved annotation diversity inevitably increase the difficulty of ensuring the annotation quality. For instance, it is much easier to annotate focusing on a few small communities than annotate across all the communities in social network. Considering these limi- tations of existing pipelines, pertinent question arises: Can design pipeline that can leverage LLMS to automatically generate high quality annotations and utilize them to train a GNN model with promising node classification performance?
METHOD
To overcome the limitations of current pipelines for node classifications, we propose a new pipeline Label-free Node Classification on Graphs with LLMs, short for LLM-GNN. It (1) adopts LLMs that demonstrate promising zero-shot performance on var- ious node classification datasets (Chen et al., [0233] He al. 2023a) as the annotators; and (2) introduces the diffieulty-aware active selection and optional filter- ing strategy to get training nodes with high annota- tion quality, representativennss, and diversity simulta- neously.
(Confidence-
(Difficulty-awar))) Active node selection
aware) Annotations
(Optiona P)Post- filtering
3.1 AN OVERVIEW OF LLM-GNN
The proposed LLM-GNN pipeline is designed with four flexible components as shown in Figure M (difficulty-aue aware) active node selection, (confidence- aware annotations), optional post-filtering, and GNN model training and prediction. Compared with the original pipelines with ground truth label, the annota- tion quality of LLM provides unique new challenge. (1) The active node selection phase is to find a candi- date node set for LLM annotation. Despite only con- sidering the diversity and representativeness (Zhang et al. 2021ch as the original baseline, we pay addi- tional attention to the influence on annotation quality. Specifically, we incorporate a difficulty-awan heuris- tic that correlates the annotation quality with the fea- ture density. (2) With the selected node set, we then utilize the strong zero-shot ability of LLMS to anno- tate those nodes with confidence-awand prompts. The confidence score associated with annotations essen- tial, LLM annotations (Chen et al. 2023), akin to human annotations, can exhibit a certain degree of label noise. This confidence score can help to identify the annotation quality and help us filter high-quality labels from noisy ones. (3) The optional post-filterigg stage unique step in pipeline, which aims to remove low-quality annotations. Building upon the annotation confidence, further refine the quality of annotations with L.MSs confidence scores and remove those nodes
GNN training
& predict
Our proposed pipeline LLM- GNN for label-free node classification graphs. It is designed with four compo- nents: (DXdifficulty-aaat lection to select suitable nodes which are easy annotate. (2) (confidence-wware an- notations) generate the annotation on the node and confidence score reflecting the label quality. (3) optional post-filtering removes low quality annotation via confi- dence score (4) GNN model training and prediction.
Figure 1:
active node
3

---

Published as a conference paper at ICLR 2024
with lower confidence from the previously selected set and (4) With the filtered high-quality notation set, we then train GNN models on selected nodes and their annotations. Ultimately, the well-trained GNN model is then utilized to perform predictions. It should be noted that the frame- work we propose is very flexible, with different designs possible for each part. For example, for the part of active node selection, we can conventional active learning methods combined with post-filtering to improve the overall quality of the labeling. We then detail each component.
3.2 DIFRE ACTIVE NODE SELECTION
Node selection aims to select a node candidate set, which will be annotated by LLMs, and then learned on GNN. Notably, the selected node set is generally small to ensure controllable money budget. Unlike traditional graph active learning which mainly takes diversity and representativeness into consideration, label quality should also be included since LLM can produce noisy labels with large variances across different groups of nodes.
In the difficulty-aware active selection stage, we have no knowledge of how LLMs would respond to those nodes. Consequently, we are required to identify some heuristics building connections to the difficulty of annotating different nodes. A preliminary investigation of LLMs' annotations brings us inspiration on how to infer the annotation difficulty through node features, where we find that the accuracy of annotations generated by LLMs is closely related to the clustering density of nodes.
To demonstrate this correlation, we employ k-means clustering on the original feature space, setting the number of clusters equal to the distinet class count. 1000 nodes sampled from the whole dataset and then annotated by LLMs. They are subsequently sorted and divided into ten equally sized groups based on their distance to the nearest cluster center. As shown in Figure 2 we observe a consistent pattern: nodes closer to cluster centers typically exhibit better annotation quality, which indicates lower annotation difficulty. Full results included in Appendix F22 We then adopt this distance as heuristic to approximate the annotation reliability. Since the cluster number equals the distinct class count, we denote this heuristic as C-Density, calculcated as C-Density(v:)
where for any node i/ and its closest cluster center CC,,, and Zv, represents the
1+ [5u, -Foc.
feature node Us We demonstrate the effectiveness of this method through theoretical explanation in Appendix K
1.00
Acturac
0.95
0.30
0.85
0.65
0.65
0.60
Region Index
Region Index (b) CITESEER
(a) CORA
Figure 2: The annotation accuracy by LLMs vs. the distance to the nearest clustering center. The bars represent the average accuracy within each selected group, while the blue line indicates the cumulative average accuracy. At group i, the blue line denotes the average accuracy of all nodes in the preceding groups.
We then incorporate this annotation difficulty heuristic in traditional active selection. For traditional active node selection,
select B nodes from the unlabeled pools with top scores (u;). where Sae() is a score function (we defer the detailed introduction of fne(() to Appendix A To benefit the performance of the trained models, the selected nodes should have trade-off between annotation difficulty and traditional active selection criteria ( (egg, representativeness (Cai et al. 2017) and di- versity (Zhang al. (021c)). In traditional graph active learning, selection criteria can usually be denoted as a score function, such as PageRank centrality fpy (1;) for measuring the structural diver- sity. Then, one feasible way to integrate difficulty heuristic into traditional graph active learning ranking aggregation. Compared to directly combining several scores via summation or multiplica- tion, ranking aggregation is more robust to scale differences since it is scale-invariant and considers only the relative ordering of items. Considering the original score function for graph active learning as faa (v:), we denote r Ffa (w;)) as the high-to-low ranking percentage. Then, we incorporate difficulty heuristic by first transforming C-Density(e:) into rank TC-Deasit (v;). Then combine these two scores: (u;) = ag x Th (v;) + aj X De- (u,). "DA" stands for difficulty aware. Hyper- parameters og and on introduced to balance annotation difficulty and traditional graph active
4

---

Published as a conference paper at ICLR 2024
learning criteria such representativeness, and diversity. Finally, nodes os with larger fiA-aa (1s) are selected for LLMs to annotate, which is denoted as Vanian Vanno-
Table 1: Accuracy of annotations. Yellow denotes the best and Green denotes the second best result.
0.90
Der
The cost is determined by comparing the token consumption to that of zero-shot prompts.
Accuracy
Hel Covd Hil Divooud
0.80
Cora
WIOICS
Prompt Stringyy
Acc (96)
Cost
Acc (5)
Cost
Acc (18)
Cost
0.70
(zero
75.33
100
150
300
Vanilla
Topk (zemo shoo
62,00 6.38
14.00
Most Voting tzero stoo
68.001 + 7.35
1.1
75.33 + 4.99
1.1
000.0 2:16
1.3
Figure 3: An illustration of the relation be- tween accuracy and confidence on WIKICS.
Hsteil (remedhot)
6.880
1.5
73.67
5.25
1.4
71.00 2.65
14
Hifest lone shot)
6::25
2.5
2.5
77.67
2,02
299
3.3 CONFIDENCE-AWARE ANNOTATIONS
After obtaining the candidate set Vanno through active selection, use LLMs generate annota- tions for nodes in the set. Despite we select nodes easily to annotate with difficulty-aware active node selection, we are not aware of how the LLM responds to the nodes at that stage. It leads to the potential of remaining low-quality nodes. To figure out the high-quality annotations, we need some guidance on their reliability, such as the confidence scores. Inspired by recent literature on generating calibrated confidence from LLMs (Xiong et al. 2023 Tian et al.) 2023 Wang et al., 2022). we investigate the following strategies: (I) directly asking for confidence TTan al. 2023), denoted as "Vanilla (zero-shot)"; (2) reasoning-based prompts to generate annotations, including chain-of-thought and multi-step reasoning (Wei et al.) [022] Xiong et al. (023); (3) TopK prompt, which asks LLMs to generate the top K possible answers and select the most probable
as the prompt (Wang et al. 2022). which queries LLMs "Most voting": (5) Hybrid prompt (Wang et al.] 2023a), which combines both TopK prompt and consistency-based prompt. In addition to the prompt strategy, few-shot samples have also been demonstrated critical to the performance of LLMs (Chen et al. 2023). We thus also investigate incorporating few-shot samples into prompts. In this work, we try 1-shot sample to avoid time and money cost. Detailed descriptions and full prompt examples are shown in Appendix D
answer (Xiong et al., 2023); (4) Consistency-baut multiple times and selects the most common output as the answer, denoted
Then, we do a comparative study identify the effective prompts in terms of accuracy, calibration of confidence, and costs. (1) For accuracy and cost evaluation, we adopt popular node classification benchmarks CORA, OGBN-PRODUCTS, and WIKICS. We randomly sample 100 nodes from each dataset and repeat with three different seeds to reduce sampling bias, and then we compare the generated annotations with the ground truth labels offered by these datasets. The cost is estimated by the number of tokens in input prompts and output contents. (2) It is important to note that the role of confidence is to assist us in identifying label reliability. Therefore, To validate the quality of the confidence produced by LLMs is to examine how the confidence can reflect the quality of the corresponding annotation. Therefore, we check how the annotation accuracy changes with the confidence. Specifically, we randomly select 300 nodes and sort them in descending order based on their confidence. Subsequently, we calculate the annotation accuracy for the top k nodes where K is varied in (50, 100, 150, 200, 250, 300). For each K, a higher accuracy indicates a better quality of the generated confidence. Empirically, we find that prasoning-basee prompts will generate outputs that don't follow the format requirement, and greatly increase the query time and costs. Therefore, we don't further consider them in this work. For other prompts, we find that for a small portion requirrss of inputs. the outputs of LLMs do not follow the format requirements (for example, outputting an annotation out of the valid label names). For those invalid outputs, we design self self-correction prompt and set a larger temperature to review previous outputs and regenerate annotations.
The evaluation of performance and cost is shown in Table I and the evaluation of confidence is shown in Figure 3 The full results are included in Appendix From the experimental results, we make the following observations: First, LLMs present promising zero-shot prediction performance on all datasets, which suggests that LLMs are potentially good annotators. Second, compared to zero-shot prompts, prompts with few-shot demonstrations could slightly increase performance with double costs. Third, zero-shot hybrid strategies present the most effective approach to extract high- quality annotations since the confidence can greatly indicate the quality of annotation. We thus adopt
5

---

Published as a conference paper at ICLR 2024
zero-shot hybrid prompt in the following studies and leave the evaluation of other prompts one future work.
3.4 POST-FILTERING
After achieving annotations together with the confidence scores, we may further refine the set of annotated nodes since we have access confidence scores generated by LLMs, which can be used to filter high-quality labels. However, directly filtering out cow-coniieence nodes may result in label distribution shift and degrade the diversity of the selected nodes, which will degrade the per- formance of subsequently trained models. Unlike traditional graph active learning methods which try to model diversity in the selection stage with criteria such as feature dissimilarity (Ren et al.) 2022), in the post-filtering stage. label distribution is readily available. As a result, consider the label diversity of selected nodes. To mcasure the change of diversity, we propose a simple score function change of entropy (COE) to measure the entropy change of labels when moving a node from the selected set. Assuming that the current selected set of nodes is Viel. then COE can be computed as: COE(#;)
directly
H ((:))(()))) (ūv...) where H () is the Shannon entropy function [Shannon] 1948). and ÿ denotes the annotations generated by LLMs. It should be noted that the value of COE may possibly be positive or negative, and small COE(u:) value indicates that removing this node could adversely affect the diversity of the selected set, potentially com- promising the performance of trained models. When a node is removed from the selected set, the entropy adjusts accordingly, necessitating a re-computation of COE. However, it introduces neg- ligible computation overhead since the size of the selected set Vanna usually much smaller than the whole dataset. COE can be further combined with confidence [conr(t;) ) to balance diversity and annotation quality, in a ranking aggregation manner. It should be noted that TC-Density (v;) is also available in the post filtering phase. So, the final filtering score function fütter can be stated as: funer( = Bu x (V))(())) + 81 x (COF(e.) + 32 X TC-Density (vi).
Su. B1, and B3 introduced to balance label diversity and annotation quality. rfan is the high-to-low ranking percentage of the confidence score scont- To conduct post-filtering, each time we remove the node with the smallest fütter value until a pre-defined maximal number is reached.
3.5
GNN TRAINING AND PREDICTION
After obtaining the training labels, we further train GNN. Our framework supports a variety of GNNs, and we select GCN, the most popular model, as our primary subject of study. Additionally, another critical component during the training process the loss function. Traditional GNN-based pipelines mainly adopt cross-entropy loss; however, due to the noisy labels generated by the LLMs, we may also utilize a weighted cross-entropy loss in this part. Specifically, we can use the confidence scores from the previous section as the corresponding weights.
4 EXPERIMENT
In this section, we present experiments to evaluate the performance of our proposed pipeline L.M-- GNN. We begin by detailing the experimental settings. Next, we investigate the following research questions: RQL. How do active selection, post-filtering, and loss function affect the performance of LLM-GNN? How to come up with an effective pipeline implementation? RQ2. How does the performance and cost of LLM-GNN compare to other label-free node classification methods? RQ3. How do different budgets affect the performance of the pipelines? RO4. How do LLMs' annotations compare to ground truth labels?
4.1 EXPERIMENTAL SETTINGS
In this paper, we adopt the following TAG datasets widely adopted for node classification: CORA (McCallum et al. 2000), CITESEER GGile al.) 1998), PUBMED (Sen et al. 2008). OGBN- ARXIV, OGBN-PRODUCTS (Hu et al. 2020b), and WIKICS (Meryyei & Cangea 2020). Statistics and descriptions of these datasets are in Appendix C
In terms of the settings for each component in the pipeline, we adopt gpt-3.5-turbo-066 to gen- erate annotations. In terms of the prompt strategy for generating annotations, we choose the "zero- shot hybrid strategy" considering its effectiveness in generating calibrated confidence. We leave the evaluation of other prompts as future works considering the massive costs. For the budget of the ac- tive selection, we refer to the popular cemi-supervised learning setting for node classifications (Yang et al. 2016) and set the budget equal to 20 multiplied by the number of classes. For GNNs, we adopt one of the most popular models GCN (Kipf & Welling 2016). The major goal of this evaluation is
httts://ppaate//ee at openai com wcom//oca/noeti /twp///aa////eeeeeoooooo/oo 8 //ppt- /gppt---5555 3-5

---

Published as a conference paper at ICLR 2024
Table 2: Impact of different active selection strategies. We show the top three performance in each dataset with pink, green, and yellow, respectively. To compare with traditional graph active selec- tions, we underline their combinations with our selection strategies if the combinations outperform their corresponding traditional graph active selection. OOT means that this method can not scale to large-scale graphs because of long execution time.
CORA
CITESEER
PUBMED
WIKICS
OGBN-ARXIY
OGBN-PRODUCTS
Random
70.48 0.73
65.11 + 1.12
72.98 + 2.15
60.69 1.73
64.59 + 0.16
70,40 +: 0.60
Random-W W
71.77+
65.92
1.05
73.92 = 1.75
61.42= 1.54
64.95 + 0.19
71.96* 0.59
C-Density
42.22 = 1.59
66.44 0.34
+
57.77 0.85
44.08 = 0.39
8.29 = 0.00
PS PS-Random-VV W
72.38 0.72
67.18 0.92
73.31 = 1.65
62.60 # 0.94
65.22 = 0.15
71.62+ 0.54
Density
72.40
61.06 0,95
74.43
64.96 0.53
$1.77 +: 0.24
20.22 + 0.11
Density-W
72.39
59.88 + 0.97
0.
63.80 0.69
51.03 + 0.27
20.97 = 0.15
DA-Density
62.92 0.32
7.243
63,08 0.45
51.33 = 0.29
8,50 0.32
PS-Density-W W
74.61+0.13
61.00 0.55
0.25
65.57 0.45
51.73 + 0.29
19.15 = 0.18
DA- DA-Density-W W
67.29
62.98
+ 0.77
73,39 0.35
63,26 0.62
51.36 + 0.39
8.52
AGE
0.38
54.25 + 0.31
74.55 +: 0.54
55.51 0.12
46.68 0.30
65.63 = 0.15
AGE-W
69.40
64.30 0.49 57.60 0.35
47,84+ 0.35
64.92 0.19
DA-AGE
74.38 0.24
59.92 0.42
74,20 0.51
59.39 + 0.21
48.21 + 0.35
60.03 = 0.11
PS-AGE-W
72.61 0.39
57.44 + 0.49
6.40
56.13 0.11
47.12 + 0.39
68.62 = 0.15
DA-AGE-W
74.96 + 0.22
58.41 + 0.45
65.85 + 0.67
59.19 + 0.24
47.794 0.32
59.95 # 0.23
RIM
69.86 0.38
63.44 + 0.42
76.22 + 0.16
66.72 0.16
OOT
OOT
DA-RIM
73.99 + 0.44
60.33 + 0.40
79.17 + 0.11
67.82+ 0.32
OOT
OOT OOT OOT
PS-RIM-W
73.19
62.85 0 0.49
74.52 + 0.19
69.84 0.19
OUT
DA-RIM-W
74.73 + 0.41
60.80 + 0.57
77.94 +: 0.24
68.22 0.25
OOT
GraphPart
66.59 + 1.34
77.50 + 1.23
67.28 0.87
OCT
OOT
GraphPart-Vy W
68.20 1.42
78.91 = 1.04
68.43 = 0.92
OOT
OOT
DA DA-GruphPart
69.35 + 1.92
69.37 1,27
79,49 + 0.85
68.72 1.01
OOT
OOT
PS GraphPart-VV W
69.924 1.75
69.06 1.19
78.84 + 1.0S
66.90 1.05
OOT
OOT
DA-GraphPart-W
68.61 + 1.32
68.82 # 1.17
79.89 + 0.79
67.13 # 1,23
OOT
OOT
FeatProp
72.82 0.08
66.61 # 0.55
73.90 = 0.15
64.08 0.12
66.06 = 0.07
74.04 = 0.15
FeatProp-W W
73.56 0.13
68.04 0.69
76.90 + 0.19
63.80 0.21
66.32 = 0.15
74.32 = 0.14
"S-FeetProp
75.54 + 0.34
69.06 0.32
74.98 + 0.35
66.09 0.35
66. 14 0.27
74,91 = 0.17
PS- PS-FeatProp-W
76.23 + 0.07
68.64 + 0.71
78.84 + 1.05
64.72 + 0.19
65.84 + 0.19
74.54 = 0.24
show the potential of the LLM-GNN pipeline. Therefore, we do not tune the hyper-parmmeters in both difficulty-waare active selection and post-filtering but simply setting them with the same value.
In terms of evaluation, we compare the generated prediction of GNNs with the ground truth labels offered in the original datasets and adopt accuracy as the metric. Similar to (Ma et al., 2022), we adopt a setting where there's no validation set, and models trained on selected nodes will be further tested based on the rest unlabeled nodes. All experiments will be repeated for 3 times with different seeds. For papee--arrmeters of the experiment, we adopt fixed setting commonly used by previous papers or benchmarks Kipf& & Welling (2016); Hamilton et al. (2017); Hu et al.) (2020b). One point that should be strengthened is the number of training epochs. Since there's no validation set and the labels are noisy, models may suffer from overfitting (Song et al.] (022). However, we find that most models work well across all datasets by setting a small fixed number of training epochs, such as 30 epochs for small and medium- datasets (CORA, CITESEER, PUBMED, and WIKICS), and 50 epochs for the rest large-scale datasets. This setting, which can be viewed as a simpler alternative to early stop trick (without validation set) (Bai et al. 2021) for training on noisy labels that we can compare different methods more fairly and conveniently.
4.2 RQ1. IMPACT OF DIFFERENT ACTIVE SELECTION STRATEGIES
We conduct a comprehensive evaluation of different active selection strategies, which is the key component of our pipeline. Specifically, we examine how effectiveness of (1) difficulty-aware active node selection before LLM annotation (2) post- filttfiiteing after LLM annotation and how they combine with traditional active learning algorithms (3) loss functions. For selection strategies, we consider
(1) Traditional graph active selection: Random selection, Density-based selection (Ma et al.
2022
GraphPart (Ma et al. 2022). FeatProp (Wu et al. 2019), Degree- selection (Ma et al.
2022
Pagerank centrality based selection (Ma et al. (022), AGE (Cai et al. (017), and RIM (Zhang al. 2021b). (2) Difficulty- active node selection: C-Density-bard selection, and traditional graph active selections combined with C-Density. To denote these selections, we add the prefix "DA."
7

---

Published as a conference paper at ICLR 2024
For example, "DA-AGE" means combining the original AGE method with our proposed C-Density. (3) Post Filtering: Traditional graph active selection combined with confidence and Caseb selections, we add the prefix "PS-". For FeatProp, as it selects candidate nodes directly using the K-Medoids algorithm (Wu et al. 2019), integrating it with difficulty-aware active selections is not feasible. For loss functions, we consider both cross entropy loss and weighted cross entropy loss, where we add "-W" postfix for the latter. Detailed introductions of these methods are shown in Appendix A The results for GCN are shown in Table 2
In terms of space limits, we move part of
the results and more ablation studies to Appendix
From the experimental results, we make the following observations:
1. The proposed post-filtering strategy presents promising effectiienness Combined with traditional graph active learning methods like GraphPart, RIM, and Featprop, can consistently outperform. Combined with FeatProp, it can achieve both promising accuracy and better scalability.
2. Although C-Density-basee
selection can achieve superior annotation quality, merely using this metric will make the trained model achieve poor performance. To better understand this phe- nomenon, we check the labels of the selected nodes. We find that the problem lies in label imbalance brought by active selection. For example, we check the selected nodes for PUBMED, and find that all annotations belong to one class. We further find that tuning the number of clus- tering centers for C-Density can trade off between diversity and annotation quality, where larger K can mitigate the class imbalance problem. However, it proposes a challenge to find a proper K for sassive-scale datasets like OGBN-PRODUCTS, where weighted loss and post-fil post-filtering more effective.
Comparing normal cross entropy loss weighted cross entropy loss, weighted cross entropy loss further enhance the performance for most of the cases.
4. In nutshell, we summarize the following empirical rules of thumbs: (1) Featprop- basedd based methods can consistently achieve promising performance across different datasets efficiently; (2) Compar- ing DA and PS, DA costs less since we don't need L.LMs to generate the confidence and we may use simpler prompt. PS can usually get better performance. On large-scale datasets, PS usually get much better results.
4.3 (RQ2.) COMPARISON WITH OTHER LABEL-FREE NODE CLASSIFICATION METHODS
To demonstrate the effectiveness and novelty of our proposed pipeline, we further conduct a comparison with other label-free node classifi- cation pipelines, which include: (1) Zero-shot node classification method: SES, TAG-Z (Li & Hooi] 2023); (2) Zero-shot classification mod- els for texts: BART-large-MNNJ (Lewis al.,
Table 3: Comparison of label-free node classifi- cation methods. The cost is computed in dollars. The performance of methods with * are taken from Li & Hooi (2023). Notably, the time cost of LLMs is proportional to the expenses.
OGBI-ARXIV
OGBN-TRODUCTS
2019]: and (3) Directly using L.MSs for predic- tions: -rredit-rre (Chen et al., 2023). Detailed introductions of these models can be found in Appendix A We compare both perfor- mance and costs of these models, and the results are shown in Table
Methods
Acc
Cost
Acc
Cost
SES(*)
13.08
N/A
6.67
N/A
TAG-Z(()
37.08
N/A
47.08
N/A
BART-large-MNLI
13,2
N/A
28.8
N/A
LLMMs-ss-reeiccoo
73.33
79
75.33
1572
LLM-GNN
66.32
0.63
74.91
0.74
From the experimental results in the table, we can see that (1) our proposed pipeline LLM-GNN can significantly outperform SES, TAG-Z and BARL-Large-MNLL (2) Despite LLMs-as-Predictors has better performance than LLM-GNN, its cost is much higher than LLM-GNN. For example, the cost of I.L.Ms-as-Predditorrs in OGBN-PRODUCTS is 2, 124x that of LLM-GNN. Besides, the promising performance of prompts leveraging the memorization of LLMs (Chen et al. 2023).
OGBN-ARXIV may be an exception, relevant to the specific
4.4 (RQ3.) How DO DIFFERENT BUDGETS AFFECT THE PERFORMANCE OF OUR PIPELINES?
We conduct a comprehensive evaluation on different budgets rather the fixed budget in previous experiments. It aims to examine how effective our algorithm is when confronting different real- world scenarios with different to meet different cost and performance requirements. Experiments are typically conducted on the CORA dataset by setting the budget as (35, 70, 105, 140, 175, 280, 560, I,120). We choose both random selections and those methods that perform well in Table 2 We can have the following observations from Figure 4 (1) with the increase in the budget, the performance tends to increase gradually. (2) unlike using ground truth, the performance growth
8

---

Published as a conference paper at ICLR 2024
is relatively limited the budget increases. It suggests that there exists trade-off between the performance and the cost in the real-world scenario.
4.5 (RQ4.) CHARACTERISTICS OF LLMS' ANNOTATIONS
Although LLMs' annotations are noisy labels, we find that they are more benign than the ground truth labels injected with synthetic noise adopted in (Zhang al.) 2021b). Specifically, assuming that the quality of LLMs' annotations is 9%, we randomly select (1 9)% ground truth labels and flip them into other classes uniformly to generate synthetic noisy labels. We then train GNN models LLMs' annotations, synthetic noisy labels, and LLMs' annotations with all incorrect labels removed, respectively. The results are demonstrated in Figure [
from which we observe that: LLMs' annotations present totally different training dynamics from synthetic noisy labels. The extent over- werr-fiting for LLMs' annotations is much less than that for synthetic noisy labels.
1.0
Filtered
Accuracy
Accuracy
65
Random
0.7
Random WE PS Random Featprop PS Featprop
60
0.5
55
35
70
105
140 175 280 560 1120 Budget
75
125
Epochs
Investigation on how different budgets affect the performance of LLM- GNN. Methods achieving top performance in Table 2 and random selection-basee meth- ods are compared in the figure.CE and WE means normal cross entropy loss and weighted loss, respectively.
Figure 4:
Figure Comparisons among LLMs' annota- tions, ground truth labels, and synthetic noisy labels. "Ran" represents random selection, "GT" indicates the ground truth labels. "Fil- tered" means replacing all wrong annotations with ground truth labels.
5 RELATED WORKS
Graph active learning, Graph active learning (Cai et al. 2017) aims to maximize the test perfor- mance with nodes actively selected under limited query budget. To achieve this goal, algorithms are developed to maximize the informativeness and diversity of the selected group of nodes (Zhang al., 2021c]. These algorithms are designed based on assumptions. In In[Ma et al.) (2022). diversity assumed to be related to the partition of nodes, and thus samples are actively selected from dif- ferent communities. In Zhang al. (202 bla), representativeness is assumed to be related to the influence of nodes, and thus nodes with a larger influence score are first selected. Another line of work directly sets the accuracy of trained models as the objective (Gao al. 2018 Hu et al. 2020a Zhang al. 2022), and then adopts reinforcement learning to do the optimization.
LLMs for graphs. Recent progress on applying LLMss for graphs (He et al. 2023a Guo aims to utilize the power of LLMs and further boost the performance of graph-related tasks. LLMs
2023)
are either adopted as the predictor (Chen et al., 2023 Wang et al. 2023al Ye et al.) 2023). which directly generates the solutions
the enhancer (He al. (023a). which takes the capability of smaller model with better efficiency. In this paper, we adopt
LLMs to boost the performance LLMs annotators, which combine the advantages of these two lines to train efficient model with promising performance and good efficiency, without the requirement of any ground truth labels.
CONCLUSION
In this paper, revisit the long-term ignorance of the data annotation process in existing node clas- sification methods and propose the pipeline label-free node classification on graphs with LLMs to solve this problem. The key design of our pipelines involves LLMs to generate confidence-ane annotations, and using difficulty- selections and confidence- based post-filtering to further en- hance the annotation quality. Comprehensive experiments validate the effectiveness of pipeline.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
This research is supported by the National Science Foundation (NSF) under grant numbers CNS 2246050, IIS1845081, IIS2212032, IIS2212144, IOS2107215, DUE 2234015, DRL 2025244 and IOS2035472, the Army Research Office (ARO) under grant number W91 INF-211--- the Home Depot, Cisco Systems Inc, Amazon Faculty Award, Johnson&Johnson, JP Morgan Faculty Award and SNAP.
REFERENCES
Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24392 244003, 2021.
Parikshit Bansal and Amit Sharma. Large language models as annotators: Enhancing generalization of nip models at minimal cost. arXiv preprint arXiv:2306.1570 2023.
Hongyun Cai, Vincent W Zheng, and Kevin Chen-Chuan Chang. Active learning for graph embed- ding. arXiv preprint arXiv: 705.05085, 2017.
Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, al. Exploring the potential of large language models (Ilms) in learning on graphs. arXiv preprint ar arXiv:2307. .03393, 2023.
Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty, and Boyang Li. gpt-3 good data annotator? arXiv preprint arXiv:2212.10450. 2022.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. survey for in-context learning. arXiv preprint arXiv:2301.00234 2022.
Li Gao, Hong Yang. Chuan Zhou, Jia Wu, Shirui Pan, and Yue Hu. Active discriminative network representation learning. In Proceedings of the Twenty-Sevent International Joint Conference Artificial Intelligence, IJCAI-18, pp. 2142-2148. International Joint Conferences on Artificial Intelligence Organization, 7 2018. doi: 10.24963/jcci..211//777 URL htttt:tt////ett org/
10. N0.249633/1jal htt.j/aaa.221/99666 2018/296
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text- annotation tasks. arXiv preprint arXiv:2303. 15056, 2023.
C. Lee Giles, Kurt D. Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing system. In Proceedings of the Third ACM Conference on Digital Libraries, DI. 98, 89-98, New York, NY. USA, 1998. ACM. ISBN 0-89791-965-3, doi: 10.1145/27667522766 URL http: acm. wtgg/1/ 10. 11445226675.276633
Jiayan Guo, Lun Du, and Hengyu Liu. Gpt4graph: Can large language models understand graph structured data? empirical evaluation and benchmarking. arXiv preprint arXiv:2305.10666 2023.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Haibo He and Edwardo A. Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 2109):1263-128. 2009. doi: 10.1109/TKEE.2088...
Xiaoxin He, Xavier Bresson, Thomas Laurent, and Bryan Hooi. Explanations as features: Llm- based features for text-attributed graphs. arXiv preprint arXiv:2305. 19523, 2023a.
Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiso, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. Annollm: Making large language models to be better crowd- sourced annotators. arXiv preprint arXiv:2303.1688 2023b.
10

---

Published as a conference paper at ICLR 2024
Cöté, Zhiyuan Liu, and Jian Tang. Graph policy network for transferable active learning on graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10174-10185. Curran Associates, Inc., 2020a. URL https::/prrocesd ngs www.ipsuri.... CC www..ppppp...p..pppppe ffles/paper/20207 T11e/73740eaBB  /ttp:///000SS0022200000000066020
Shengding Hu, Zheng Xiong. Meng Qu, Xingdi Yuan,
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 18-22133, 2020b.
Jin Huang, Xingjian Zhang, Qiaozhu Mei, and Jiaqi Ma. Can Ilms effectively leverage graph struc- tural information: When and why. arXiv preprint arXiv:2309.16595 2023.
Sheng-jun Huang, Rong Jin, and Zhi-Hua Zhou. Active learning by querying informative and representative examples. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010. URL httpsss/pprroeeddgg..rrrooo.... Paper.
files/
Wei Ju, Yifang Qin, Siyu Yi, Zhengyang Mao, Kangjie Zheng, Luchen Liu, Xiao Luo, and Ming Zhang. Zero-shot node classification with graph contrastive embedding network. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
net/
Forum?1d=BwCH JRLSy.
Thomas N Kipf and Max Welling. Supi-super classification with graph convolutional net- 2016.
works. arXiv preprint
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequencc pre- training for natural language generation, translation, and comprehension. CoRR, 13411 2019. URL http: / aawii.ivvvv..... www.xxx..og/aa//////e /11.0.91.... 13461
Yuexin Li and Bryan Hooi. Prompt-based zero-and few-shot node classification: A multimodal approach. arXiv preprint arXiv:2307.11572 2023.
Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. arXiv preprint arXiv:2310.00149, 2023.
Jiaqi Ma, Ziqiao Ma, Joyce Chai, and Qiaozhu Mei. Partition-based active learning for graph neural networks. arXiv preprint arXiv:2201 20019.991, 2022.
Yao Ma and Jiliang Tang. Deep learning on graphs. Cambridge University Press, 2021.
Andrew McCallum, Kamal Nigam, Jason D. M. Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127-163, 2000.
Péter Mernyei and Cătälina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural net-
works. arXiv preprint arXiv:2007.0290 2020.
Nicholas Pangakis, Samuel Wolken, and Neil Fasching. Automated annotation with generative ai requires validation. arXiv preprint arXiv:2306.00117 2023.
Xiaohuan Pei, Yanxi Li, and Chang Xu. Gpt self-supervision for a better data annotator. arxiv preprint wtXiv:23006.044999 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert- networks. In Conference on Empirical Methods in Natural Language Processing, 2019. URL :001 6309
https://api iementicsccho www.aarrrrrroooooooooeeee hww.aattccccco....rr/r//rppppppe
Zhicheng Ren, Yifu Yuan, Yuxin Wu, Xiaxuan Gao, Yewen Wang, and Yizhou Sun. Dissimilar nodes improve graph active learning. arXiv preprint arXiv:2212.01968 2022.

---

Published as a conference paper at ICLR 2024
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al Magazine, 29(3):93, Sep. 2008 doi: 10. 10.1609/aimag..229322222 URLpattps://o//t aaai wwt.saaaaaiiaaaaaaaaaaaaa httpaa//r//.mmaaaaeieeeeeeeeeeee php/ a
C. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27
(3):379-423, 1948. doi: 10.1002/j.15533 1512-73333---
7308.-730.1144 73305.998.1b01133
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and lae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions Neural Networks and Learning Systems, 2022.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint 1.Ki:22305.14955. 2023.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks, arXiv preprint 1.Xiv:1710.10900 2017.
Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? arXiv preprint arXiv:2305.10037, 2023a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308, 1432, 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-consisenncy improves chain thought reasoning in language models. arXiv preprint arXiv:2203. 11171, 2022.
Zheng Wang, Jialong Wang, Yuchen Guo, and Zhiguo Gong. Zero-shot node classification with decomposed graph prototype network. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1769-1779, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of thouuht prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu. and Yang Liu. Learn- ing with noisy labels revisited: A study using real-world human annotations. arXiv preprint 1xXi:22110.12088 2021.
Yuexin Wu, Yichong Xu, Aarti Singh, Yiming Yang, and Artur Dubrawski. Active learning for graph neural networks via node feature propagation. arXiv preprint arXiv:1910.07560 2019.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306. 13063, 2023.
Zhilin Yang. William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. ArXiv, 000.0660...8.1, 2016. URL httpss:/app.
semanticscholar. www.hhoorrrrrrrrraooooooooooeeee Catgg/CorpssI:: 70:778752
Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Natural language is all graph needs. arXiv preprint arXiv:2308 07 134, 2023.
Qin Yue, Jiye Liang, Junbiao Cui, and Liang Bai. Dual bidirectional graph convolutional networks for zero-shot node classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, pp. 2418-2417, New York, NY, USA. 2022. Association for Computing Machinery, ISBN 978 4503938550 doi: 10.1145/3534678.55333
U htt.:http:///omm //tttss://oiio www.:orggg.... wttg./...14//353313 3539316
12

---

Published as a conference paper at ICLR 2024
Wentao Zhang, Yu Shen, Yang Li, Lei Chen, Zhi Yang, and Bin Cui. Alg: Fast and accurate active learning framework for graph convolutional networks. In Proceedings of the 2021 International Conference Management of Data, SIGMOD '21, pp. 2366-2374, New York, NY. USA, 2021a. Association for Computing Machinery. ISBN 9781 450383431. doi: 10.1 00.1145/34480163357725 UtR.https:///dt http://toooooo/..../ 1145/ 801 3457325
Wentao Zhang, Yexin Wang. Zhenbang You, Meng Cao, Ping Huang, Jiulong Shan, Zhi Yang, and Bin Cui. Rim: Reliable influence-based active learning on graphs. Advances in Neural Information Processing Systems, 34:27978-27990, 2021b.
Wentao Zhang, Zhi Yang, Yexin Wang, Yu Shen, Yang Li, Liang Wang, and Bin Cui. Grain: Im- proving data efficiency of graph neural networks via diversified influence maximization. arXiv preprint arXiv:2108 wtt.:2108.0022. 002199, 2021c.
Yuheng Zhang, Hanghang Tong, Yinglong Xia, Yan Zhu, Yuejie Chi, and Lei Ying. Batch active learning with graph neural networks via multi-agent deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9118 9-962, 2022.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations (ICLR 2023), 2023.
Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space. arXiv preprint arXiv:2310.0108, 2023.
Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances Neural Information Processing Systems, 01227955-27977 2021a.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. In International Conference on Machine Learning, pp. 12912-12923. PMLR, 2021b.
13

---

Published as a conference paper at ICLR 2024
A DETAILED INTRODUCTIONS OF BASELINE MODELS
In this part, we present more detailed introduction to related works, especially those baseline models adopted in this paper.
A.1 GRAPII ACTIVE LEARNING METHOSS
1.
Density-based selection (Ma et al. 2022): We apply KMeans clustering in the feature space with the number of clusters set to the number of budgets. Then, nodes with closest distances to their clustering centers are selected the training nodes. fact is calculated as the reciprocal of the distance to the clustering centers.
GraphPart (Ma al., 2022): A graph partition method is first applied to ensure the selection diversity. Then inside each region, nodes with high aggregated density are selected based on an approximate K-medoids algorithm. fact() is calculated as the reciprocal of the distance to the local clustering centers after partition.
3. FeatProp (Wu et al., 2019): Node features are first aggregated based on the adjacency matrix. Then, K-Medoids is applied to select the training nodes. fact () is calculated using K-Medoids.
4. Degree-based selection (Ma et al., 2022): Nodes with the highest degrees are selected as the training nodes. Sac(() is calculated the degree od nodes.
5. Pagerank-baze based selection (Ma et al., 2022): Nodes with the highest PageRank centrality are selected as the training nodes. face calculated as the PageRank score of nodes.
6. AGE (Cai et al., 2017): Nodes are selected based on a score function that considers both the density of features and PageRank centrality. We ignore the uncertainty measurement since we adopt one-step selection setting. Taer() is calculated as the weighted aggregation of the density of aggregated features together with the PageRank score.
7. RIM (Zhang et al., 202Tb); Nodes with the highest reliable social influence are selected as the training nodes. Reliability is measured based on a pre-defined oracle accuracy and similarity between the aggregated node feature matrices. fuce() is based on the density of aggregated features.
A.2 ZERO-SHOT CLASSIFICATION BASELINE MODELS
SES (Li & Hooi| 2023 y: An unsupervised method to do classification. Both node features and
class names are projected into the same semantic space, and the class with the smallest distance selected as the label.
2. TAG-Z (Li & Hooi 2023);; Adopting prompts and graph topology to produce preliminary logits,
which are readily applicable for zero-shot node classification.
3. BART-Laare-MNNIJ (Lewis et al. 2019): A zero-shot text classification model fine-tuned on the MNLI dataset.
B MORE RELATED WORKS
LLMs Annotators. Curating human-annotated data is both labor-intensive and expensive, pecially for intricate tasks or niche domains where data might be scarce. Due to the exceptional zero-shot inference capabilities of LLMs, recent literature has embraced their use for generating
pseudo-annotated data (Bansal & Sharma 2023 Ding et al. [022 Gilardi et al., 2023 He et al.,
2023b Pangakis et al., 2023 Pei al.) 2023). Gilardi et al. (2023): Ding al. (2022) evaluate
I.LMs* efficacy annotators, showcasing superior annotation quality and cost- cost-efficency compared
to human annotators, particularly in tasks like text classification. Furthermore, |Pei et al. (2023);
He al. (202bby; Bansal & Sharma (2023) delve into enhancing the efficacy of LLM-ggnerated annotations: Pei et al. (2023); He et al. (2023b) explore prompt strategies, while [Bansal Sharma (2023) investigates sample selection. However, while acknowledging their effectiveness, Pangakis ct al. (2023) highlights certain shortcomings in LLM--produeed annotations, underscoring the need for caution when leveraging LLMS in this capacity.
Zero-shot Node Classification. Another issue related to work is zero-shot node classifica- tion. The goal of zero-shot node classification is to train a model on existing training data and then generalize to new categories. It is important to note that this is different from the concept of label-
14

---

Published as a conference paper at ICLR 2024
free node classification that propose, in problem there labels available from the start. Waang et al. (2021) transfers the knowledge from existing labels to unseen labels by semantic descriptions of labels. Yue et al. (2022) adopts label consistency module to transfer the knowl- edge from the original label domain to the unseen label domain. Ju et al. (2023) adopts a two-level
contrastive learning to learn the node embeddings and class assignments in end-to-end manner,
which can thus transfer the knowledge to unseen classes.
C DATASETS
In this paper, we use the following popular datasets commonly adopted for node classifications: CORA, CITESEER, PUBMED, WIKICS, OGBN-ARXIV, and OGBN-PRODUCTS, Since LLMs can only understand the raw text attributes, for CORA, CITESEER, PUBMED, OGBN-ARXIV, and OGEN- PROdUCTS, we adopt the text attributed graph version from Chen al. (2023). For WIKISS, we get the raw attributes from ht / //githhb..... com/ Somernyeeiwwikie cS- cs-datasett Then, we give a detailed description of each dataset Table
Table 4: Dataset descriptions
Duttset Name
#Nodes
#Edges
Taak Description
Clasess
CORA
2708
5429
predict the category of this
Given the cidl and abstract.
Thenry, Reinforement Learning, Prohuhilistic Methols
Rule Lezening Neunl Networks
Case Based, Geneckc Algoritmss,
CITESEER
S180
4277
Given the rite and abstract,
predict the
this
Human Computer Inseraction, Artiiciaa Intelligence
Agonts, Machine I Leminng Information Revievall Dambose,
Given the cile abstract. canegory paper
19717
Diabetes Mellinus Experimetaal,
Mellits Type
Type
Giivnn the
Computationl linguisties,
Wikipedin artile.
Datuhsees Operating systemss Computer acchttectuee
215863
category
Cumputer security.
Compuner file systems,
Distibuted computing archilecture, sechnology. Programming language topics
OCON-ARXIV
Given
abstract, prodict the category paper
go www.iii..ogg..........o.o wwt.b//www///cc/c//////oo
OORN PRODUCTS 2440029
61830140
prodict the category of this product
Given product description,
classes from Amazon, including Home Kitchen, Ilealth & Personal Care...
D PROMPTS
In this part, show the prompts designed for annotations. We require LLMs output Python dictionary-like object so that it's easier to extract the results from the output texts. We find that LLMs sometimes generate information without following the users' instructions. As result, design the self-correction prompt to fix those invalid outputs (Table
Table 5: Prompt used for self-correction
Previous prompt: (Previous input)
Your previous output doesn't 't follow the format, please correct it old output: (Previous output)
Your previous answer (Previous answer) is not valid class.
Your should only output categories from the following list: (Lists of label names)
New output here:
We then demonstrate two concrete examples of our prompts. The first prompt is based on the zero- shot consistency strategy (see Table 6). The second prompt is based the few-shot consistency strategy (see Table 77). It should be noted that the latter prompt is only used to be compared with zero-shot prompts. We don't 't use it in Section 4 For one-shot prompts, we leverage large language models to automatically generate both the top predictions and confidence scores according to the philosophy of Zhang al. (2023)
15

---

Published as a conference paper at ICLR 2024
Table 6: Full prompt example for zero-shot annotation with consistency strategy on the PUBMED dataset
Input: Question: (Contents) Paper: Title: Heritability of pancreatic beta-cell function among nondiabetic members of Caucasian familial type diabetic kindreds. Abstract: Both defective sulin secretion and insulin resistance have been reported in relatives of type 2 diabetic subjects. We tested 120 members of families with type diabetic sibling pair with a Aebutamide-modiitie frequently sampled i.v... Task:
There following categories:
[diabetes mellitus experimental, diabetes mellitus type 1, diabetes mellitus type 2]
What's the category of this paper?
Provide your best guesses and confidence number that each is correct (0 to 100) for the following question from most probable least. The sum of all confidence should be 100. For example, ("answer": <your first answer>>, "confidence": <confidence for first answer>>).... Output:
Table 7: Full prompt example for 1-shot annotation with TopK strategy on the CITESEER dataset
Input: I will first give you an example and you should complete task following the example. (Contents for in-context learning samples) Argument in Multi-Agent Systems Multi- agent systems .Skippeee for clarity) Computational societies are developed for two primary rea- sons: Mode...
Question:
Task:
There are following categories: lagents, machine learning, information retrieval, database, human computer interaction, artificial intelligence]
What's the category of this paper?
Provide your 3 best guesses and confidence number that each is correct (0 to 100) for the follow- ing question from most probable least. The sum of all confidence should be 100. For example, [ ""answer": <your answer>, "confidence"; <confidence for first answer> Output:
[["answer": "agents", "confidence"; 60), ("answer": "artificial intelligence", "confidence". 30),
("answer": "human computer interaction", "confidence": 10]]
Question: Decomposition in Data Mining: An Industrial Case Study Data (Contents for data to be annotated)
Task:
There are following categories: [agents, machine learning, information retrieval, database, human computer interaction, artificial intelligence] What's the category of this paper? Provide your best guesses (The same instruction, skipped for clarity) ...
Output:
E COMPLETE RESULTS FOR THE PRELIMINARY STUDY ON EFFECTIVENESS OF CONFIDENCE-AWAN PROMPTS
The complete results for the preliminary study on confidence-aware prompts are shown in Table Figure 6 and Figure 7
16

---

two specific Published as a conference paper at ICLR 2024
Table Accuracy of annotations generated by LLMs after applying various kinds of prompt strate- gies. We use yellow to denote the best and green to denote the second best result. The cost is determined by comparing the token consumption that of zero-shot prompts.
CoRA
CITESBER
PUBMED
OGRN-ARAIV
OCEN-PRODUCTS
WIKICS
Prompt Streegy
Apc (8)
Cost
Acc (16)
Cost
Acc (1)
Cost
Acc (R)
Cost
Acc: (8)
Cost
Auc (9)
Cost
Zono-sht
68.33 6.55
64.00
7.79
88.67
2.62
1
73.67 4.1.9
1
68.33 1.89
One-shot
65.67
90.67
1:25
2.0
78.67
18
72,00
Topk
66.00
5.10
Mosd Voting
68 00 7.35
1.1
65.33 7.41
88.33 2.49
1.4
73:33
75.33 4:99
1.1
69100
2.16
Hybrid (Zero-shot)
67.33 6.880
1.5
66.35 6.24
1.5
87.33
0.94
1.7
72.33
4.19
1.2
73.67
5.25
14
71.00
2.83
Hyhrid (One-shor)
70.35 6.24
2.9
67.67
+ 7.41
2.7
90.
2.2
70,00 + 2.16
2.1
15.07
6.13
2.3
73.67
2.62
2.9
One Sho-Shhot
Most Voting
Top-K
Hybrid(One- Shot)
Zero-Shot
HybridjZero-Shue
WkICS
0.90
0,90
Accuracy
Accuracy
0.85
0.80
0.80
0.70
0.75
100
50
250
300
Figure 6: Relationship between the group accuracies sorted by confidence generated by LLMs. refers to the number of nodes selected in the groups.
F PRELIMINARY OBSERVATIONS OF LLM'S ANNOTATIONS
F.1 LABEL-LEVEL OBSERVATIONS
We begin by examining the label-Ievel characteristics of annotations generated by LLMs. Label distribution is critical as it can influence model training. For example, imbalanced training labels may make models overfit to the majority classes (He & Garcia 2009). Specifically, we juxtapose the distribution of LLM annotations against the ground truths. To facilitate this analysis, we employ noise transition matrices, a tool frequently used study both real-worke noisy labels (Wei et al.) 2021) and synthetic ones (Zhu et al. 2021b). Considering classification problem with N classes, the noise transition matrix will an N x N matrix where each entry Tiy represents the probability that instance from the true class i is given the label j. If i then Tip is the probability that a true class i is correctly labeled as i. If i * j. then Tip is the probability that a true class is mislabeled j.
To generate the noise transition matrices, we sample 1000 nodes randomly for each dataset. Fig- ure & demonstrates the noise transition matrix for WIKICS, CORA, and CITESEER, three datasets with proper number of classes for visualization. We demonstrate the results of adopting the "zero- shot hybrid" prompting strategy in this figure. We illustrate the results for few-shot demonstration strategies and synthetic noisy labels in Appendix F.I Since the number of samples in each class imbalanced, we also show the label distribution for both ground truth and LLM's annotations in Figure
An examination of Figure & reveals intriguing patterns. FIRST, the quality of annotations, defined as the proportion of annotations aligned with the ground truth, exhibits significant variation across different classes. For instance, in the WIKICS dataset, LLMs produce perfect annotations for sam- ples where the ground truth class is co (referring to "Computational linguistics"). In contrast, only 31% of annotations are accurate for samples belonging to 67, referring to "distributed computing architecture". Similar phenomena can be observed in the CORA and CITESEER datasets. SECONd, for those classes with low annotation quality, incorrect annotations will not be distributed to other classes with uniform probability. Instead, they more likely to fall into categories.
For example, for WIKICS, C ("distributed computing architecture") tends to flip into es ("web technology"). The reason may be that these two classes present similar meanings, and for some samples, categorizing them into any of these two classes is reasonable. Moreover, find that such
17

---

Published as a conference paper at ICLR 2024
One-Shot
Most Voting
-
Top-K
Hybrid(One-Shha
Zero-Shot
Hyerid(Zero-Sho)
0.80
2.00
Acaurary
0.97
0.70
0.68
0.90
0.80
0,90
Figure 7: Relationship between the group accuracies sorted by confidence generated by LLMs. refers to the number of nodes selected the groups
kind of flipping can be asymmetric. Although e7 in WIKICS tends to flip into ca, samples from Os never flip into c7. These properties make LLMs" annotations highly different from the synthetic noisy labels commonly adopted in previous literature (Song et al. 2022). which are much more uniform among different classes.
5:27
9:15
E.AL
5/2
0.22
AUP
5.29
5/6
0229
211
SM
315
8:00
3:5
8:33
8/2
4.4L
6:3
0.3
9:14
S/4
5.4L
CA4
WOX
S/W
ABC
2/2
8.27
5.1
3/4
KM
MM
XXC
PM
5
9:11
111
1:0
$
6
7
3
5
(a) WIKICS
(b) CORA
(c) CITESEER
Figure 8: Noise transition matrix plot for WIKICS, CORA, and CITESEER with annotations gener- ated by LLMs
2:27
0:11
3.11
2:23
E12
2.31
N
2.37
1.22
8/6
1,77
2
WAY
S:N
GM
122
9:4
8.5
SUW
km
10
5.0
LG
631
6.3
3
5
$
1
(a) WIKICS
(b) CORA
(c) CITESEER
Figure 9: Noise transition matrix plot for WIKICS, CORA, and CITESEER with annotations gener- ated by LLMs using prompts with few-shot demonstrations
Referring to Figure we observe noticeable divergence between the annotation distribution generated by LLMs and the original ground truth distribution across various datasets. Notably, in the WIKICS dataset, what is originally minority class, Cs emerges of the majority classes. similar shift is evident with class cg in the CORA dataset. This trend can largely attributed to the asymmetry inherent in the noise transition matrix.
18

---

Published as a conference paper at ICLR 2024
2.1
1:27
5:14
0.88
9.4
531
601
5.3
5.11
1.21
SAT
2:18
5:0
0.e
0.1
8:92
8/2
0.75
OAR
AGL
9/6
622
5.77
S21
5:18
948
903
6.2
8:M
6.29
2.31
1:21
2:31
6.31
662
5-A
2
3
5
6
0
2
6
5
(a) WIKICS
(b) CORA
(c) CITESEER
Figure 10: Noise transition matrix plot for WIKICS, CORA, and CITESEER with annotations gener- ated by injecting random noise into the ground truth.
#Appearances
Ground truth
#Appearances
Ground truth Armotaation
300
Arestiations
200
200
100
100
0
1
2
3
8
9
2
3 Labels
5
6
Labels
(a) WIKICS
(b) CORA
Figure 11: Label distributions for ground truth labels and LLM's annotations
F.2 RELATIONSHIP BETWEEN ANNOTATION QUALITY AND C-DENSITY
As shown in Figure 12 the phenomenon on PUBMED is not as evident as on the other datasets. One possible reason is that the annotation quality of L.MM on PUBMED is exceptionally high, making the differences between different groups less distinct. Another possible explanation, as pointed out in (Chen et al. 2023), is that LLM might utilize some shortcuts present in the text attributes during its annotation process, and thus the correlation between features and annotation qualities weakened.
1.00
1.00
[verage Recuracy
0.85
0.95
0.90
0.90
0.85
0.85
0.80
0.80
0.75
0.75
0.70
0.70
0.65
0.65
0.60
0.0
Region Index
Region Index
(a) WIKICS
(b) PUBMED
Figure 12: Relationship between the group accuracies and distances the clustering centers. The bar shows the average accuracy inside the selected group. The line shows the accumulated average accuracy.
19

---

Published as a conference paper at ICLR 2024
G EFFECTIVENESS OF CONFIDENCE GENERATED BY LLMS
In this section, we demonstrate the effectiveness of confidence generated by LLMs with a case study the Cora dataset. We plot the confidence calibration plot with three prompt strategies: zero-shot, TopK, and hybrid. From the results, we can see that hybrid prompt can generate accurate and diverse confidence scores, which is more effective.
08
64
Accurcyy
0.8
Accurscy
0.5
0.2
0.4
0.8
1.8
8.
0.2
6.8
0 68
1.0
(a) Zero-shot
(b) TopK
(c) Hybrid
Figure 13: Comparison of three prompt strategies
H HYPERPARAMTERS
We now demonstrate the hyper-parameters adopted this paper, which inspired by Hu et al. (020b):
1. For small-scale datasets including CORA, CITESEER, PUBMED, and WIKICS, we set: learning rate 0.01, weight decay 5 hidden dimension 64, dropout to 0.5.
2. For large-scale datasets including OGBN-ARXIV and DGBN-PRODUCTS, we set: learning rate to 0.01, weight decay to 5e hidden dimension 256, dropout 0.5.
1 COMPARING LLMS ANNOTATIONS TO SYNTHETIC NOISY LABELS
1.0
0.9
Accurocy
0.8
Accuraoy
0.7
0.7
0.5
50
125
75
150
Epochs
Epochs
(a) CORA
(b) CITESBER
Figure 14: The red represents the performance of models trained with ground truth labels. The yellow curve represents the performance models trained with LLMs annotations with all wrong labels fixed. The blue curve represents the models trained by LLMs' annotations. The green curve represents the models trained by synthetic noisy labels with the same annotation quality as LLMss annotations. The solid line represents the performance on the test set, while the dashed line represents the performance on the training set.
20

---

Published as a conference paper at ICLR 2024
J EXTRA RESULTS FOR THE COMPARATIVE STUDY IN TABLE
In Table 10 we fürther demonstrate the results for combing weighted loss, difficulty-aware selection, and post-filtering to degree-based selection and pagerank-bol based selection. In Table we explore the effectiveness of applying PS and DA together. We find that applying them simultaneously usually won't 't get good performance, which means it's probably not the proper way to integrate LLMs' confidence into selection. As comparison, enhance the performance most of the time. Even though it can not surpass the baselines, the gap is very small, which shows its effectiveness.
find that using weighted cross-entropy loss can
Table 9: Ablation study for the effectiveness of different combinations.
Cora
CiteSeer
AGE
69.15 0.38
54.25 0.31
DA-AGE
74.38 = 0.24
59.92 0.42
DA-AGE-W
74.96 0.22
58.41 0.45
PS-DA-AGE
56.38 + 0.14
RIM
69.86
63.44
DA-RIM
73.99 0.44
60.33 0.40
DA-RIM-W
74.73 0.41
60.80
PS-DA-RIM
72.34
60.33 0.40
Table 10: Extra results for degree-based -aased selection and pagerank-based selection
CoRA
CITESEER
PUBMED
WIKICS
OGBN-ARXIV
OGBN-PRODUCTS
Degree
68.67 & 0.30
60.23 0.54
67.77
0.07
65.38
0.35
$4.98 + 0.37
71.22 =: 0.21
Degree-W W
69.86 0.35
60.47
68.24
65.61 + 0.31
55.69 + 0.24
71.96 + 0.23
DA-Degree
72.86 = 0.27
0.52
0.04
63.40 0.51
55.32 0.33
44.41 = 0.53
PS. PS-Degree-W W
70.92 = 0.28
0.69
67.21 = 0.29
55.89 + 0.39
71.57=
DA DA-Degree-W W
73.01 # 0.24
61.29
74.11
63.14 0.55
55.35 = 0.32
47.90
Pagerank
70.31 0.42
68.58 0.11
67.13 0.46
59.52 0.03
69.20 0.32
Pagerank-V W
7
68.86 0.19
69.61 0.34
$9.60 + 0.04
69.75 0.29
DA DA- Pagerank
74.34 +: 0.41
0.40
7
67.15 0.44
58.82 + 0.52
$4.77 = 0.36
PS- Pagerank-W
74.81 a: 0.37
63.27
68.23 +
69.86 0.29
58.84 a: 0.14
69.69 + 0.45
DA- HA-Pagerank-W
75.62 0.39
61.25 0.45
73.60 0.22
68.19
59.40 0.26
55.57 = 0.24
K THEORETICAL MOTIVATION
In this section, we further analyze the theoretical motivation for difficulty-aware selection. In nutshell, our objective is show why C-Density is a useful metric to select nodes with high annotation quality.
Given the parameter distribution of LLMs Q, the parameter distribution of encoder P (SBERT), we assume ground truth Y2 R NMM, pseudo label Y E N*M, and node features encoded by the encoder X E xxd Here, M denotes the number of classes and denotes the hidden dimension. Our objective is to maximize the accuracy of annotations, which is thus to minimize the discrepancies between and Yz.
min =x) - Yr) = Σ (0....)R))
where (m.,..., ny the index of the selected nodes, and [(Y)= defined as follows: :h represents the feature of node n4. while xr); represents the unknown latent embedding of node n4
&
- = Σ
= 1n(1
Σ a 13) = In(1 + | xz
21

---

Published as a conference paper at ICLR 2024
Since we only have access to the node features X generated by encoder 8, need to make connection between Q and P.
Lemma For any annotation y generated by LLMs, Ev-of(y) Σ log Bp exp(f(v)) + (T)
Proof.
Egr= = 1 1()p(()d s(1) p(u)
( = Ea ef)e P(y)
((1)
q(y)
Since KI(Q]IP) = R log 9(v) p(u)
log ( log Rg o (1) p(y) ((v)
2 E log f(y) [(u)((y)
-Ba- log (I(2(P)
q(y)
Then, transform the objective into
min Ev exp(f(Y)) (g1)).
Assuming the label distribution H. we further have min By>=BBrra exp([())))()))))))))))))))))))>) min (mi.....n)))
By- Xt.11
(R
Assuming X and XG. follows gaussian distribution, where X ((i,)))) Xt, ~ N N(uj, oz), then
E (IXX xx113) = E [I(X p4i) - (X2 pg)) + (1+ - (1;))))))
E ||X mil²) + E (IXx, - (1x1²) + Iless 13l1
= non + noz + ||m - mill
Since (uz, оz) is unknown, μi is fixed, thus we want to minimize os Given an arbitrary node, a, can be viewed as the distance to the clustering centers. The smaller σ; is, the smaller the corresponding E. (IIX- - {x) also hecomes, indicating that the minimum value of Ea (fy) is attained when a, is at its smallest. This demonstrates why nodes closer to the clustering centers are "preferred" by LLMs and thus achieve better annotation quality.
L DESIGN PHILOSOPHY BEHIND LLMGNN
Regarding LLMGNN, we have the following two key designs:
1. In the annotation process, do not consider structural information, hence we have designed a structure-free prompt. 2. In the choice of a LLM, we do not use the more advanced GPT-4, but instead, choose the designs are currently the most appropriate because: 1. We discover that using structure-aware prompt does not effectively improve annotation quality without introducing ground truth labels, due to the limited structural understanding capabilities of LLMs. 2. We find that compared to GPT-3.5- turbo, the improvement brought by GPT-4 is very limited. After exploration, we realize that this is related to the ambiguity in the ground truth labels of the node classification task. Currently, GNN is the best tool capable of utilizing structural information to handle ambiguity.
cost-effective GPT-3.5-turbo. In practice, find that these two
To begin with, assigning correct labels for node classification is hard for LLMs. We show the annotation quality (the ratio of annotations matching the ground truth labels) of GPT3.5 and GPT4 in Table II by randomly selecting the annotated samples (140 nodes for Cora, and 120 nodes for CiteSeer, and we control the seed to make sure we select the same set of nodes). We can see that GPT4 doesn't 't give much better annotations (in terms of accuracy) than GPT3.5. The overall performance (less than 70%) indicates that node classification is not an easy task for LLMs.
If further check the annotation results given by these two different models, find that one common bottleneck preventing these models from getting better results is the annotation bias (also can be viewed as "label ambiguity") of the labels. We showcase example in Table 12
22

---

Published as a conference paper at ICLR 2024
Table 11: Comparison GPT-3.5 and GPT-4 on the annotation task
CORA
CITESEER
GPT3.5
67.62
2.05
66.39 8.62
GPT4 68.81 + 1.87
65.56 9.29 9.29
Table 12: An example from the CITESEER dataset
Attribute of one node from Citeseer: "Discovering Web Access Patterns and Trends by Applying OLAP and Data Mining Technology on Web Logs As confluence of data mining and WWW technologies, it is now possible to perform data mining on web log records collected from the Internet web page access history...
Input:
The ground truth label for this node is "Database", while the prediction of both two L.MMs "Infor- mation Retrieval". From human beings' understanding, both "Database" and "Information retrieval" are somewhat reasonable categories of this node. However, because of the single label setting of node classification, only one of them is correct. This somehow demonstrates that to get high accu- racy on node classification, the models need to capture such kinds of dataset-specific bias in the annotation (the information in LLMs is more like "commonsense knowledge", so there mismatch). Structural information may help us find such kind of bias. For example, if the category of neighboring nodes is more easier predict, and most of them should be related to "Database". then this node is more likely to come from "Database". This phenomenon has also been observed in Chen al. (2023).
be
So, why LLMs with structure-aware prompts can not improve their performance by introducing that structural information (shown in Table |3 It's mainly because of LLMs' poor understanding capability of structural information. Huang et al. (2023) and Wang et al. 2023a try various kinds of structure-awar prompts, and find that LLMs present limited structure reasoning abilities with current prompt designs. Chen et al. (2023) and Huang et al. (202)) further show that incorporating neighboring ground truth labels can improve performance. However, ground truth labels are not available in the label-free settings. As comparison, we find that merely summarizing the neighboring contents may not effectively improve the performance. Under the current structure- aware prompt design, LLMs only utilize the structural information in very limited manner (like using neighboring labels). As a comparison, message-passe structural information effectively and efficiently. For example, we try replacing GNN with MLP in LLMGNN in the Table [4] and we observe a huge performance gap.
GNNs can capture complicated
Table 13: Comparison of using structure-aware and structure-free prompts
CORA
CORA
CITESEER
CITESEER
No Struct
Struct
No Struct
Struct 65.92 + 0.43
FeatProp
72.82 = 0.08
69.08 + 0.39
66.61 0.55
FeatProp + PS
75.54 + 0.34
67.55 - 0.70
69.06 0.32
67.80 + 0.45
Based on the above reasons, we only use textual information in the labeling process, while in the training process of GNN, we utilize structural information. At the current stage, we believe this to be a more effective paradigm. This highlights the motivation for to design LLMGNN which can enjoy the advantages both LLMs and GNNs while mitigating their limitations.
Designing effective structure-aware prompt is valuable future direction, and methods like Agent- prompting (Wang et al. 2023b) (multi-round prompt) may help to further improve the performance. The main focus of our paper is to propose flexible framework that supports var- ious kinds of prompt designs, and new prompt designs can also enhance the effectiveness of our framework.
23

---

Published as a conference paper at ICLR 2024
Table 14: Comparisons of LLMss-s-Predictorr LLMGNN, and LLMMLP, which demonstrates the superiority of GNN
LLMssas-Prediccos
LLMGNN
LLMMLP
CORA 68.33
76.23
67.06
The importance the prompt is reflected in the fact that if we can improve the quality of annotations, further enhance the effectiveness of LLMGNN. For example, if we correct portion of incorrect annotations to the right ones, we may improve the performance of LLMGNN, The "original quality" means the original annotation given by LLMs. "+k%" means that randomly turn "k%" wrong annotations into correct ones. The results are shown Table 15
Table 15:
The performance-changiin trend of Random selection and PS-FeatProp- W selection when fix portion of the wrong annotations
original quality
+7%
+12%
+17%
Random
70.48
72.66
75.92
78.58
PS- PS-FeatProp-VV W
76.23
77.95
80.25
81.96
Moreover, we want show that the annotation quality will influence the scaling behavior of LLMGNN when we increase the budget. From Table 16] (the first row means the budget equal l number classes we can see that (1) When the budget low, difference between LLMGNN trained by high-quality and low-quality annotations is smaller (< 20); when the budget high, this difference becomes larger(> 20); (2) the performance of LLMGNN is closely related to Cora). With
the annotation quality (the performance of LLMGNN on PubMed is better than
better annotation quality, LLMGNN may potentially achieve a higher performance upper bound when gradually increase the budgets.
Table 16: The scaling behavior LLMGNN on CORA and PUBMED when we increase the budgets
10
15
20
25
40
80
160
CORA
57.35
66.45
68.42
70.17
69.64
70.68
72.07
72.73
PURMED
62.49
64.56
72.44
73.16
75.92
77.8
82.38
82.5
24


Reviews:
[{"review_id": "6f2Z4aNkfX", "reviewer": "Reviewer_ugc7", "summary": "This paper aims to combine Large Language Models(LLMs) and Graph Neural Networks(GNNs), leveraging their strengths. GNNs achieve promising performance when dealing with graph-structured data, while it needs abundant high-quality labels to ensure the performance. On the other hand, LLMs shows impressive zero-shot proficiency on text-attributed graphs, but it suffers from high inference costs and processing structural data. This paper suggests LLM-GNN, using LLM for annotation, providing training signals on GNN for further prediction. Moreover, the authors propose node selection strategy and confidence-aware annotation for efficient learning with high quality annotations.", "questions": "- In Figure 2 or Figure 12, the trend of decreasing accuracy as the distance between nodes and cluster centers increases seems somewhat weak in average accuracy. Even in Table 2, when DA is added to traditional graph active selection and when DA is added to the use of PS, there are many cases where performance actually decreases. I acknowledge that tuning was not performed, but there are still too many cases where performance declines. While the authors said that grid search would improve the performance in a specific case, it seems necessary to perform more tuning across a broader range of cases to clearly demonstrate the effectiveness of DA.\n- In Table 6, is it realistic to use labels in 1-shot example when we are considering the \"Label-free\" setting? Furthermore, how are the confidence scores determined in that example? More detailed explanation is required.\n- In Figure 4, why do some methods show an increase in performance after a budget of 70, followed by a sharp decline? Could this be attributed to class-imbalance issues during the selection process?\n- While there is an example in Figure 5, it would be beneficial to demonstrate on different datasets that LLM-GNN achieves competitive performance with GNNs trained on ground truth labeled data, while significantly reducing costs.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is well-motivated.\n- Each component supports motivation reasonably.\n- Well written paper, it is easy to follow.", "weaknesses": "- The performance gain is incremental, especially for Difficulty-aware active node selection (DA).\n- Explanations about experiments are not enough, and some parts are unclear. Please refer to questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qpFCDvj1cs", "reviewer": "Reviewer_JunD", "summary": "Given a text attributed graph without labels, the paper proposes a cost-effective method that combines LLMs and GNNs to annotate the labels in four steps. In the first step, the paper selects the nodes that should be annotated and terms it  as difficulty aware selection. It combines active learning (selection) techniques along with a difficulty score which is based on the distance from the center of a cluster. In the second step, annotations are created for the selected nodes using an LLM which also generates a confidence score. In the third step, nodes are pruned such that the confidence score of LLMs is high without significantly changing the diversity of nodes (via change in entropy). Finally, a GNN is trained on the graph to generate labels for other nodes.", "questions": "- Post-filtering (PS): How useful are the confidence scores generated by the LLMs (Appendix F.1, table 6)? Showing the mean and variance of confidence score for each dataset may help in understanding its impact on performance.\n\n- Providing the number of nodes selected in each step in the experiment will also help in understanding the effects of the steps. For instance, could you please provide the number of selected nodes during active selection and DA-active selection? Also, what is the pruning ratio when applying PS? All this information can help in understanding the efficacy of the steps.\n\n- PS-DA-methods have not performed well in most cases compared to DA-methods (Table 2). Any insights on this can help in understanding these steps better.\n\nMinor Questions:\n\n- Detail explanation on f_{act}(vi) is missing\n\n- It is mentioned in page 5 that the detailed descriptions and full prompt examples are shown in Appendix D. However, it is missing any detailed descriptions.\n\nTypos:\n\n- Page 7: (4) Combing - supposed to be combining?\n\n- Page 3: GNN modelson- models on (space missing)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of label-free annotation using LLMs on text attributed graphs is an interesting research direction introduced by the paper.\n\n- The paper has experimented with different datasets and incorporated various existing techniques to come up with a cost-effective model.\n\n- This paper appropriately balanced traditional graph active selection criteria with annotation quality by incorporating difficulty-aware active selection with post filtering to obtain training nodes from LLM.", "weaknesses": "- Difficulty aware (DA) selection: According to the paper, LLMs annotation quality degrades when they have to annotate nodes which are away from the centers. It implies that the LLMs annotation quality would suffer in case of the diverse nodes (away from center).  However, GNNs accuracy will only improve if the nodes are diverse. Hence, difficulty aware selection i.e. use of c-density might not always help and, in fact, it may hinder in some cases. This is also evident from the results shown in Table 2 : Active_Selection_Methods and the corresponding  DA-Active_Selection_Methods show similar performance on average across different techniques (i.e., selection methods). Moreover, for at least 50% of the cases, the DA-method (row 2) performs poorly compared to the corresponding active learning method (row 1). \n\n- Though the accuracy from the LLM-GNN model is good (and of course, the model is efficient), it couldn't outperform LLM as a predictor (Table 3).\n\n- The methods are heuristics and do not have theoretical evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7SQadl2a8U", "reviewer": "Reviewer_HZKD", "summary": "In this paper, the authors study label-free node classification by combining LLMs with GNNs. Specifically, the proposed method first leverages LLMs to annotate a small portion of nodes, then uses GNNs with the pseudo-labels to classify the remaining large portion of nodes. The main challenges lie in how to actively select nodes and leverage LLMs to obtain reliable labels for those nodes. Three modules, including difficulty-aware active node selection, confidence-aware annotations, and post-filtering are proposed to tackle the challenges. Experimental results on text-attributed graphs demonstrate the effectiveness of the proposed method, especially compared with heuristically choosing annotated nodes.", "questions": "See Weaknesses above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is among the first trials of combining LLMs with GNNs to solve a novel problem, i.e., label-free node classification.\n2. The proposed method is clearly described and the paper is easy to follow in general.\n3. The authors compare with various heuristic baselines and conduct analyses to demonstrate the efficacy of the proposed method.", "weaknesses": "1.\tThough I acknowledge that the proposed method is a valid solution, the technical contribution of the paper is somewhat limited, especially considering that the three major components are largely based on heuristic observations, and the rest are based on existing LLMs and GNNs. It would make the paper stronger if some theoretical analyses could be provided for the proposed components.  \n2.\tThe authors should more explicitly mention that their proposed method only works for text-attributed graphs rather than any general graph, e.g., in the abstract and introduction. Otherwise, the paper may have overclaiming issues.  \n3.\tIn generating the initial node labels using LLMs, it seems that only the feature information is utilized and no structure is considered. Since it is well-known in the graph machine learning literature that both features and structures greatly affect the node labels, there exists a large room for improvement.    \n4.\tThere are some missing related works regarding zero-shot node classification such as [1-2], which should be added.  \n5.\tI also wonder how different LLMs affect the model (the reported results are all based on GPT-3.5-turbo).   \n\n[1] Zero-shot Node Classification with Decomposed Graph Prototype Network, KDD’21  \n[2] Dual Bidirectional Graph Convolutional Networks for Zero-shot Node Classification, KDD’22", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c2ESuNg2pC", "reviewer": "Reviewer_LGrX", "summary": "The paper introduces a novel approach called LLM-GNN for label-free node classification on graphs, which combines the strengths of Graph Neural Networks (GNNs) and Large Language Models (LLMs) while mitigating their limitations. It addresses the challenge of obtaining high-quality labels for graph-structured data by leveraging LLMs' zero-shot learning capabilities. LLM-GNN actively selects nodes for annotation by LLMs, generates confidence-aware annotations, and refines annotation quality through post-filtering. The approach achieves impressive results on a massive-scale dataset, OGBN-PRODUCTS, without the need for costly human annotations.", "questions": "see the weakness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. LLM-GNN presents an innovative approach to node classification on graphs by harnessing the complementary strengths of GNNs and LLMs. It acknowledges the challenges of obtaining high-quality labels and proposes a label-free solution, which is a significant contribution to the field of machine learning.\n2. The paper demonstrates the cost-effectiveness of LLM-GNN by achieving high accuracy on a large dataset with annotation costs under 1 dollar. This cost-efficient approach is particularly relevant for real-world applications with resource constraints.\n3. LLM-GNN offers a comprehensive methodology that not only utilizes LLMs for annotations but also considers active node selection, confidence-aware annotations, and post-filtering. This approach ensures the quality, representativeness, and diversity of annotations, addressing key challenges in label-free node classification.", "weaknesses": "1. Since LLMs generate annotations without access to ground truth labels, there is a risk of noisy annotations.  It would be better to investigate the robustness of LLM-GNN to noisy annotations and potential strategies for mitigating their effects.\n2.  LLM-GNN's performance is demonstrated on a specific dataset (OGBN-PRODUCTS), and while it achieves impressive results, its generalizability to other datasets or domains is not thoroughly explored in the paper. The effectiveness of the approach in different scenarios and with various types of graphs should be investigated to assess its broader applicability.\n3. There could be better with a detired comparison on the economic perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)