Title: Prompt-based Node Feature Extractor for Few-shot Learning on Text-Attributed Graph
Venue: NeurIPS 2023
Accepted: no
Keywords: Text-attributed graph; graph neural network; language model
Abstract: Text-attributed Graphs (TAGs) are commonly found in the real world, such as social networks and citation networks, which consist of nodes represented by textual descriptions. 
Currently, mainstream machine learning methods on TAGs involve a two-stage modeling approach: (1) unsupervised node feature extraction with pre-trained language models (PLMs); and (2) supervised learning using Graph Neural Networks (GNNs). 
However, we observe that these representations, which have undergone large-scale pre-training, do not significantly improve performance with limited amount of training samples. 
The main issue is that existing methods have not effectively integrated information from the graph and downstream tasks simultaneously. 
First, G-Prompt introduces a learnable GNN layer (i.e., adaptor) at the end of PLMs, which is fine-tuned to better capture the masked tokens considering graph neighborhood information.
After the adapter is trained, G-Prompt incorporates task-specific prompts to obtain interpretable node representations for the downstream task.
Our experiment results demonstrate that our proposed method outperforms current state-of-the-art (SOTA) methods on few-shot node classification. More importantly, in zero-shot settings, the G-Prompt embeddings can not only provide better task interpretability than vanilla PLMs
but also achieve comparable performance with fully-supervised baselines.

Abstract

Text-attributed Graphs (TAGs) are commonly found in the real world, such as social networks and citation networks, and consist of nodes represented by textual descriptions. Currently, mainstream machine learning methods on TAGs involve a two-stage modeling approach: (1) unsupervised node feature extraction with pre-trained language models (PLMs); and (2) supervised learning using Graph Neural Networks (GNNs). However, we observe that these representations, which have undergone large-scale pre-training, do not significantly improve performance with a limited amount of training samples. The main issue is that existing methods have not effectively integrated information from the graph and downstream tasks simultaneously. In this paper, we propose a novel framework called G-Prompt, which combines a graph adapter and task-specific prompts to extract node features. First, G-Prompt introduces a learnable GNN layer ( i.e., adaptor) at the end of PLMs, which is fine-tuned to better capture the masked tokens considering graph neighborhood information. After the adapter is trained, G-Prompt incorporates task-specific prompts to obtain interpretable node representations for the downstream task. Our experiment results demonstrate that our proposed method outperforms current state-of-the-art (SOTA) methods on few-shot node classification. More importantly, in zero-shot settings, the G-Prompt embeddings can not only provide better task interpretability than vanilla PLMs but also achieve comparable performance with fully-supervised baselines.

1 Introduction

Text-Attributed Graphs (TAGs) are a type of graph that have textual data as node attributes. These types of graphs are prevalent in the real world, such as in citation networks [12] where the node attribute is the paper's abstract. TAGs have diverse potential applications, including paper classification [3] and user profiling[14]. However, studying TAGs presents a significant challenge: how to model the intricate interplay between graph structures and textual features. This issue has been extensively explored in several fields, including natural language processing, information extraction, and graph representation learning.

An idealized approach involves combining pre-trained language models (PLMs) [10, 20] with graph 29

neural networks and jointly training them [35, 24]. Nevertheless, this method requires fine-tuning the 30 PLMs, which demands substantial computational resources. Additionally, trained models are hard to 31

be reused in other tasks because finetuning PLM may bring catastrophic forgetting[2]. 32

Therefore, a more commonly used and efficient approach is the two-stage process [32, 34, 23]: 33

utilizing pre-trained language models (PLMs) for unsupervised modeling of the nodes' textual 34

features. (2) supervised learning using Graph Neural Networks (GNNs). Compared to joint training 35

of PLMs and GNNs, this approach offers several advantages in practical applications. For example, 36

it can be combined with numerous GNN frameworks or PLMs, and this approach does not require 37 fine-tuning PLMs for every downstream task. However, PLMs are unable to fully leverage the 38 wealth of information contained in the graph structure, which represents significant information. 39 To overcome these limitations, some works propose self-supervised fine-tuning PLMs using graph 40 information to extract graph-aware node features [3]. Such methods have achieved significant success 41 across various benchmark datasets[12]. 42

43

44

45

46

47

48

49

50

51

52

53

However, both self-supervised methods and using language models directly to process TAG suffer from a fundamental drawback. They cannot incorporate downstream task information, which results in identical representations being generated for all downstream tasks. This is evidently counterintuitive as the required information may vary for different tasks. For example, height is useful information in predicting a user's weight but fails to accurately predict age. This issue can be resolved by utilizing task-specific prompts combined with language models [26] to extract downstream taskrelated representations. For example, suppose we have a paper's abstract { Abstract } in a citation network, and the task is to classify the subject of the paper. We can add some prompts to a node's sentence: { This, is, a, paper, of, [ mask ] , subject, its, abstract, is, : , Abstract } . And then use the embedding corresponding to the [mask] token generated by PLMs as the node feature. Yet this approach fails to effectively integrate graph information.

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

To better integrate task-specific information and knowledge within graph structure, this paper proposes a novel framework called G-Prompt. G-Prompt combines a graph adapter and task-specific prompts to extract node features. Specifically, G-Prompt contains a graph adapter that helps PLMs become aware of graph structures. This graph adapter is self-supervised and trained by fill-mask tasks on specific TAGs. G-Prompt then incorporates task-specific prompts to obtain interpretable node representations for downstream tasks.

We conduct extensive experiments on three real-world datasets in the domains of few-shot and zeroshot learning, in order to demonstrate the effectiveness of our proposed method. The results of our experiments show that G-Prompt achieves state-of-the-art performance in few-shot learning, with an average improvement of avg. 4.1% compared to the best baseline. Besides, our G-Prompt embeddings are also highly robust in zero-shot settings, outperforming PLMs by avg. 2.7%. Furthermore, we conduct an analysis of the representations generated by G-Prompt and found that they have high interpretability with respect to task performance.

2 Background

2.1 Text-Attributed Graph

Let G = { V, A } denotes a text-attributed graph (TAG), where V is the node set and A is the adjacency matrix. Each node i ∈ V is associated with a sentence S i = { s i, 0 , s i, 1 , ..., s i, | S i | } , which represents the textual feature of the node. In most cases, the first token in each sentence (i.e., s i, 0 ) is [ cls ] , indicating the beginning of the sentence. This paper focuses on how to unsupervised extract high-quality node features on TAGs for various downstream tasks.

2.2 Pretrained Language Models

Before we introduce G-Prompt, we require some basic concepts of pre-trained language models.

Framework of PLMs . A PLM consists of a multi-layer transformer encoder that takes a sentence S i as input and outputs the hidden states of each token:

where h i,k is the dense hidden state of s i,k .

Pretraining of PLMs . The fill-mask task is commonly used to pre-train PLMs [4, 20, 10]. Given a sentence S i , the mask stage involves randomly selecting some tokens and replacing them with either [ mask ] or random tokens, resulting in a modified sentence ˆ S i = { s i, 0 , s i, 1 , ..., ˆ s i,k , ..., s i, | S i | } , where ˆ s i,k represents the masked token. In the filling stage, ˆ S i is passed through the transformer encoder, which outputs the hidden states of each token. We denote the hidden state of the masked token ˆ s i,k as ˆ h i,k , which is used to predict the ID of the masked token:

104

105

106

[This]

[adapter]

[is]

[a]

[good]

[method]

[SEP]

[CLS]

[This]

[adapter]

[is]

[a]

[good]

[method]

[SEP]

[CLS]

Paper A

Paper B

Paper C

Fill-mask Loss

[This]

[mothod]

Fill-mask Loss

[It]

[MASK]

[node]

[feature]

[well]

[SEP]

Paper A

Paper B

Paper C

Filter

Tokens

[extract]:

[clean]:

[change]:

Task-specific

node feature

Scores:

0.7

0.1

0.2

Romdom mask

Combine with prompts

Task: which

paper can

extract node

feature?

(a) training stage

(b) prompting stage

where f LM is a linear transformation with softmax fuction, ˆ y i,k ∈ N 1 × T , and T is the size of the 85 vocabulary. The loss function of the fill-mask task is defined as L = CE(ˆ y i , k , y i , k ) , where y i,k is the 86 ID of the masked token, and CE( · , · ) is the cross-entropy loss. 87

Sentence Embedding . The hidden state of the [ cls ] token ( h i, 0 ) and the mean-pooling of all hidden 88 states are commonly used as sentence embeddings [28, 6]. 89

Prompting on PLMs . Sentence embedding and token embedding are simultaneously pre-trained 90 in many PLMs. However, due to the gap between pretraining tasks and downstream tasks, sentence 91 embedding always requires fine-tuning for specific tasks. To address this issue, some studies 92 utilize prompts to extract sentence features [13]. For example, suppose we have a paper's abstract 93 { Abstract } , and the task is to classify the subject of it. We can add some prompts to the sentence: 94

Then this sentence is encoded by PLMs, and we let h i | p denote the hidden state of the [ mask ] token 95 in prompts. Extensive experiment shows that using prompts can shorten the gap between PLMs and 96 downstream tasks and maximize the utilization of the knowledge PLMs learned during pretraining. 97

98

2.3 Graph Neural Networks

Graph Neural Networks (GNNs) have achieved remarkable success in modeling graph-structured 99 data[30, 7]. The message-passing framework is a commonly used architecture of GNN. At a high 100 level, GNNs take a set of node features X 0 and an adjacency matrix A as input and iteratively capture 101 neighbors' information via message-passing. More specifically, for a given node i ∈ V , each layer of 102 message-passing can be expressed as: 103

where Pool {·} is an aggregation function that combines the features of neighboring nodes, such as mean-pooling. And N i denotes the set of neighbors of node i .

3 Method: G-Prompt

Utilizing the information of downstream tasks and graphs is crucial for generating high-quality 107 node representations. The term 'high quality' is inherently task-specific, as exemplified by the 108 fact that height is a useful feature in predicting user weight but fails to accurately predict age. 109 Besides, the valuable topological information of TAGs can significantly enhance the understanding 110 of textual features in TAGs. However, extracting node features using both task and graph information 111 simultaneously is significantly challenging. Current PLMs used for handling textual features are 112 graph-free, while current graph-based methods employed to extract node features are primarily 113 task-free. Therefore, this paper proposes a novel self-supervised method, G-Prompt, capable of 114 extracting task-specific and graph-aware node representations. 115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

3.1 Overview

While previous works have frequently employed PLMs to process TAGs, these investigations have been constrained in extracting a broad node representation from the text-based characteristics and have not incorporated task-specific prior knowledge. Consequently, additional learning supervision via GNNs is needed to enable the effective adaptation of these node representations to downstream tasks. To address this limitation, the paper suggests incorporating prompts and PLMs into the process of extracting task-relevant node features from TAGs. Nevertheless, PLMs only utilize contextual information to generate the prompts-related output, which may be insufficient for handling TAGs. Graph structures often contain essential information that can facilitate a better understanding of textual features. For instance, in a citation network, a masked sentence such as 'This paper focuses on [MASK] learning in AI domain' could have multiple candidate tokens based solely on context. However, if many papers related to graphs are cited, we can infer with greater confidence that the masked token is likely 'graph' . At present, PLMs operate solely based on context, and their structure is graph-free. Directly incorporating graph information into PLMs by prompts is not feasible because limited prompts cannot describe the entire topological structure adequately.

Therefore, the proposed G-Prompt leverages a self-supervised based graph adapter and prompts to make PLMs aware of the graph information and downstream task. Given a specific TAG, the pipeline of G-Prompt is as follows: (1) Training an adapter on the given TAG to make PLMs graph-aware. Specifically, we propose a graph adapter that operates on the prediction layer of PLMs to assist in capturing graph information, which is fine-tuned by the fill-mask task based on the textual data contained by the given TAG. (2) Employing task-specific prompts and fine-tuned graph adapters to generate task-aware and graph-aware node features.

3.2 Fine-Tuning PLMs with the Graph Adapter

Using adapters to enable PLMs to perceive graph information is a straightforward idea. However, unlike adapters used for downstream task fine-tuning [11, 18], the graph adapter is used to combine prompts in order to extract task-relevant node representations. This is an unsupervised process, which means that the graph adapter only receives self-supervised training on given TAGs. Consequently, the most challenging aspect of graph adapters is how to assist PLMs in perceiving graph information while also maintaining their contextual understanding capability. Additionally, the graph adapter is only trained on a given TAG, generalizing to prompt tokens can also be quite difficult. Next, we introduce the graph adapter and discuss how it overcomes these challenges in detail.

Context-friendly adapter placement. The fill-mask task involves two modules of PLMs: a transformer-based module that models context information to obtain representations of masked tokens and a linear transformation that decodes the representation to output the probable IDs of the masked token. To avoid compromising the contextual modeling ability of PLMs, the Graph Adapter only perform on the last layer of PLMs. More specifically, the graph adapter is a GNN structure combing with the pre-trained final layer of the PLMs. Given a specific masked token ˆ s i,k , The inputs of the Graph Adapter are the masked token ˆ h i,k , sentence representations of node i and its neighbors and output is the prediction of the IDs' of the masked token. This process aligns with intuition inferring a possible token based on context first and then determining the final token based on graph information. Formally,

where the z i and z j denote the sentence embedding of node i and j . Note, sentence embedding is task-free and only used to model nodes' influence on their neighbor. In this paper, we utilize sentence embedding of nodes' textual features as their node feature. Θ is all trainable parameters of the Graph Adapter.

Prompting-friendly network structure . The parameters of the adapter are only trained on the fill-mask task based on the textual data contained by the target TAG. But the adapter will be used for combining prompts to generate task-related node features in various subsequent downstream tasks. So the generalization ability of the adapter is crucial. On the one hand, the distribution of hidden states responding to masked tokens in prompts may be different from the hidden states used to train the adapter. On the other hand, the candidate tokens for task-specific prompts may not appear in the tokens of the TAG. Therefore, we carefully design the network structure of the graph adapter and utilize the pre-trained prediction layer of PLM to improve the generalization ability of it.

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

When it comes to the graph adapter's training stage, it's possible that the hidden states associated with certain prompts may not be present. This means that directly manipulating those hidden states could result in overfitting on the tokens already present in the TAGs. Therefore, the graph adapter models the influence of each modeled node on the distribution of surrounding neighbor tokens based on node feature, which remains unchanged when prompts are added. Considering that some tokens can be predicted well based solely on their context and that different neighbors may have different influences on the same node, the impact of a neighbor on a token is determined jointly by a gate mechanism and the token's context. Give a specific node i , it's neighbor j , and hidden states of a masked token ˆ h i,j ,

where a ij = sigmoid(( z i W q )( z j W k ) T ) . Here, g ( · ) represents multi-layer perceptions (MLPs) with parameters Θ g that model the influence of node j . It is worth noting that when considering the entire graph, g ( z j , Θ g ) will be combined with many marked tokens of node j 's neighbors, which can help to prevent g ( z j , Θ g ) from being overfitted on a few tokens.

Subsequently, the graph adapter combines all neighbor influence to infer the final prediction result. Since the prediction layer of PLM (i.e., f LM ( · ) ) is well-trained on massive tokens, it also contains an amount of knowledge. Therefore, the graph adapter reuses this layer to predict the final result.

In this equation, the Pool ( · ) used in this paper is mean-pooling. It is worth noting that the position of f LM ( · ) can be interchanged with pooling since it is just a linear transformation. All trainable parameters in the graph adapter are denoted by Θ = { Θ g , W q , W k } .

3.3 Model optimization of G-Prompt

The graph adapter is optimized by the original fill-mask loss, L i,k = CE(˜ y i,k , y i,k ) , where ˆ y i,k is the predicted probability of the k -th masked token for the node i and y i,k is the true label. We aim to minimize L i,k with respect to Θ .

However, in actual optimization, the prediction results of ˜ y i,k,j = f LM ( ˜ h i,k,j ) consist of many small values because of the large vocabulary size of the language model. Therefore, using mean-pooling presents a significant problem as it is insensitive to these small values. For example, during some stages of the optimization process, a node may have mostly 0 . 9 predictions for the ground truth based on each edge, with only a few being 0 . 1 . Averaging them together would result in a very smooth loss, making it difficult to train the influence of neighbors with temporarily predicted values of 0.1. To address this issue, we use geometric mean instead of mean-pooling in the finetuning stage of the graph adapter, which is more sensitive to small values. It is easy to prove that the geometric mean of positive numbers is smaller than the arithmetic means, making it harder to smooth and helping the model converge faster. formally, in finetuning stage, the loss function is:

On the right-hand side of the equation, we are essentially minimizing ˜ y i,k,j through the cross-entropy loss L i,k,j = 1 |N i | CE(˜ y i,k,j , y i,k ) . It is worth noting that the graph adapter is only performed on the last layer of PLMs. As a result, we can sample a set of masked tokens and preserve their hidden states inferred by the PLMs before training. This implies that training of graph adapters can be achieved with very few computing resources.

3.4 Prompt-based Node Representations

After training the graph adapter, it can be combined with task-specific prompts to generate taskspecific and graph-aware node representations. Similar to other prompt-based approaches, we simply add task-specific prompts directly into the textual feature. For example, we might use the prompt 'This is a [MASK] user, consider their profile: [textual feature].' Formally, this process can be expressed as ˆ h i | p = PLM ( { [ P 0 ] , [ P 1 ] ... [ MASK ] , S i } ) . Where, ˆ h i | p represents the hidden state of the inserted [MASK], while [ P 0 ] , [ P 1 ] ... refers to the task-specific prompts. The resulting hidden state is then fed into the graph encoder to decode the most probable token.

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

ˆ y i | p is a | D | -dimensional vector, where | D | is the size of the PLM vocabulary. Therefore, directly using this prediction result for node representation is not conducive to downstream tasks and storage. Thus, we use the filtered results as node features, denoted by x i | p = Filter(ˆ y i | p ) . Note, each dimension represents the probability of a token being inferred by PLMs with the graph adapter based on node textual features, neighbors' information, and task-respected prompts. Intuitively, tokens that are unrelated to downstream tasks are almost the same for all nodes. Therefore, for Y p ∈ N | V |×| D | , which denotes prediction results of all nodes. This paper sorts all columns of Y p in descending order of standard deviation and keeps the top M columns as the node features. Note, we can also manually select task-relevant tokens based on prior knowledge of the task and use them as node features.

4 Experiment

4.1 Experiment setup

Dataset. We conduct experiments on three public and real-world datasets, which are Ogbn-arxiv[12] (shorted as Arxiv), Instagram[14], and Reddit 1 , to evaluate the effectiveness of the proposed method G-Prompt. Specifically, Ogbn-arxiv is a citation network where edges represent citation relationships, nodes represent papers and the text attribute is the abstracts of papers. The task is to predict paper subjects. Instagram is a social network where edges represent following relationships, nodes represent users, and the prediction task is to classify commercial users and normal users in this network. The text attribute is the users' profile. Reddit is also a social network where each node denotes a user, the node features are the content of users' historically published subreddits, and edges denote whether two users have replied to each other. The prediction task is to classify whether a user is in the top 50% popular (average score of all subreddits). Table 1 shows detailed statistics of these datasets. More details about Instagram and Reddit are provided in the Appendix.

Compared methods. We compare the proposed G-Prompt with PLM-based and Graph-based node feature-extracting methods. For the PLM-based methods, we consider three options: (1) direct use of sentence embedding as node features, and (2) use of the hidden states of masked tokens based on hard prompts as node features. (3) use of the prediction result of masked tokens based on prompts as node feature. For graph-based methods, we compare our proposed method with GAE and GIANT, which first conduct self-supervised learning on graphs to train PLMs or node feature encoders. To ensure a fair comparison, we add prompts into graph-based baselines. Except for GAINT and OGB features, the PLM we use in this paper is RoBERTa-Large[20]. Note that all prompts used in baselines are the same as those in G-Prompt.

Implementation details. For G-Prompt, we first train three graph adapters of G-Prompt on Arxiv, Instagram, and Reddit with 50 epochs, 100 epochs, and 100 epochs respectively. All of them are optimized using AdamW[21] with warm-up. For more details on the hyper-parameter settings, please refer to the Appendix. For each node, we replace 10% tokens with [mask] and use these masked tokens to train the graph adapter. During the whole training stage, all task-related prompts are invisible. Then we use prompts, finetuned graph adapters, and PLMs to jointly extract node features. For graph-based methods, we train them on each dataset with searched hyper-parameters.

4.2 Few-shot learning

To evaluate the performance of representations generated by different methods in few-shot learning, we compare the performance of different representations at different shot numbers based on the same GNNbackbone. The GNN backbone used in the performance comparison on different shot numbers is GraphSAGE[30]. In addition, we also compare the performance of different representations combined with three different neural network architectures (i.e., MLP, and RevGAT[17]) on downstream tasks

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

with the same number of shots. For Arxiv, we use a publicly available partitioned test set, while for Instagram and Reddit, we randomly sample 60% and 33% of the data as the test sets, respectively. To consider the randomness of partitioning and training, each experimental result is based on five random partitions (the partitions are the same for different baselines), the experiment is repeated five times for each partition, and the variance of 5 × 5 results is reported.

The experiment results on different shots-num are shown in Table 2. The experiment shows that: (1) Graph-aware can improve the performance of node representation . In general, approaches that use sentence representations or those that involve self-supervised training with graph information tend to outperform non-trained representations. For example, GAE shows an average improvement of avg. 6.2% compared to RoBERTa's [cls], and GIANT shows avg. 6.2% improvement over cls representation. For graph-based self-supervised tasks, fine-tuning language models might be more suitable for larger datasets. GIANT outperforms GAE by avg. 3.0% on Arxiv, but lags behind by avg. 1.4% on Instagram and Reddit. (2) Downstream task-related prompts can improve performance for all models . For graph-free language models, prompt-based representations can improve performance by avg. 5.7%, and the overall performance of prediction values and hidden states corresponding to prompts is similar. For graph-based methods, prompts in GAE improve performance by avg. 1.3%, while prompts in GIANT lead to an average improvement of avg. 1.2%. However, we note that prompts are unstable for graph-based pre-trained models. GAE shows a decline in 4 experiments, while prompts only bring a slight improvement in GIANT (compared to language models). (3) Our method is capable of utilizing both graph perception and downstream task prompts simultaneously , achieving state-of-the-art performance. Compared to PLM representations without prompts, our method improves by avg. 10.6%. Compared to PLM-prompt, it improves by avg. 4.6%, and compared to GIANT, it improves by avg. 4.1%.

Besides, as Figure 2 shows, the node representation extracted by G-Prompt in different GNNbackbone also achieves the SOTA performance compared to other baseline methods.

(a) Arxiv

(b) Instagram

(c) Reddit

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

4.3 In-depth analysis of G-Prompt

To validate the rationality of G-Prompt, we conduct experiments to compare the performance of G-Prompt and its variants. These variants include removing the gate mechanism in graph-adapter (denoted as 'w/o gate'), keeping only self-loops while removing the input graph (denoted as 'w/o graph'), and not training graph-adapter by self-supervised learning (denoted as 'w/o SSL'). The experimental results show that all variants perform worse than G-Prompt. Specifically, removing the Graph-Adapter training process leads to avg. 2.8% decrease in performance, which demonstrates the effectiveness of training graph-adapter through the fill- mask task. After removing the graph input, the performance of G-Prompt decreases by avg. 3.8%, which further confirms that the improvement provided by G-Prompt, compared to using language model prompts directly, stems from the graph adapter's ability to assist language models in comprehending graph structures. Moreover, removing the gate mechanism results in a avg. 1.8% decrease in performance, indicating that the design of the graph-adapter structure is reasonable.

4.4 Zero-shot node classification and interpretability

The node features generated through GPrompt represent the probability of each possible word for nodes given task-related prompts, where each dimension corresponds to a specific word. This probability generation incorporates prior knowledge from PLMs, graph information, and node context. Two natural questions arise: How much knowledge is contained within this word probability? Whether the node feature can help us interpret the downstream task? Therefore, we further conduct zero-shot node classification experiments on node representations. Meanwhile, we conduct a case study on Instagram.

Zero-shot node classification. We select different sets of candidate words and sum up the probabilities of each word in the set to obtain the prediction result for node classification. We employ the ROC as the evaluation metric to assess the performance of node classification. For simplicity, ArXiv dataset only selects two categories, 'Artificial Intelligence' and 'Linguistics and Language'. The other two datasets remain unchanged. We select completely random, bag-of-words, RoBERTa-base, and RoBERTa-large as baselines, using the same prompts as G-Prompt for PLMs. We provide experiment results of G-Prompt based on RoBERTa-base (Ours-B) and RoBERTa-large (Ours-L).

According to the results shown in Table 3. (1) The bag-of-words method has almost no predictive ability. (2)The PLM through Prompts has predictive ability on different tasks (improvement compared to BOW by avg. 13%). But there is a performance difference between base and large even with the same prompt due to the sensitivity of language models to prompts [22]. (3) Compared to a language model, G-Prompt shows significant performance improvement. Specifically, G-Prompt-base improved avg. 2.7% compared to the language model. However, it should be noted that the basic predictive ability of the language model and G-Prompt are correlated. Specifically, the correlation coefficient between the results of GPrompt-L and LM-L is 0.64, while the correlation coefficient with LM-B is 0.84. (4) Moreover, selecting more candidate words through prior knowledge can effectively help G-Prompt improve its zero-shot capability, with an average improvement of avg. 4.8% for the base and avg. 5.3% for the large. However, there is no significant improvement for language models and bag-of-words. Surprisingly, by adding a small number of candidate words, G-Prompt's zero-shot performance is already close to or even sometimes surpasses supervised training with 100 shots. This result indicates that combining language models and graphs for zero-shot learning on TAG is feasible.

Interpretability. The task on Instagram is to determine whether a node is a commercial user. We use the probability corresponding to each token as the prediction value, calculate its corresponding ROC of prediction performance, and then display the top 7 tokens with the highest scores. For comparison, we also show the scores of tokens corresponding to RoBERTa-Large under the same prompt. Overall, the top 7 tokens given by our model have considerably higher ROC scores than RoBERTa-Large resulting in avg. 7.0% improvement. Additionally, our results are intuitive and can even help explain the task, for example, 'premium.' Based on this result, we search and find that there are 'premium creator subscriptions' on Instagram, which means 'Users can set their own prices and earn extra cash each month,' 2 and this information is indeed related to commercial activity. Similarly, 'niche' is also a word related to Instagram business behavior.

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

5 Related work

Modeling TAGs involves numerous works related to the NLP domain and Graph domain. Currently, pre-trained language models are the primary method for modeling the textual information in text-asgraphs [25]. Presently, pre-trained language models are mainly based on transformer structures[29], with a variety of pre-training methods, such as fill-mask [4], paragraph prediction[4], adversarial learning[10], and auto-regressive learning[27]. Based on these tasks, many excellent pre-trained models have emerged, including BERT[4], RoBERTa[20], and GPT3[1]. PLMs contain an amount of knowledge acquired through extensive pre-training data[31]. Recently, using prompts has been proposed to better utilize the performance of pre-trained language models[1]. Based on this finding, prompt learning[19, 8, 16] has achieved impressive results in few-shot and zero-shot learning and has been widely applied by other domains. Currently, the structural information in modeling TAGs is primarily modeled through GNNs, such as GraphSAGE[9], GAT[30], APPNP[7, 5] and RevGAT[17], and there are also many pre-training tasks on graphs such as GAE[15], GraphCL[33] that can be extended to TAGs. Recently, many methods explore better utilizing the knowledge of PLMs to model TAGs more effectively, such as pre-training language models through graph-related tasks [3] and finetuning PLMs together with GNNs via knowledge distillation[24] or variational inference [35].

6 Conclusion

This paper proposes G-Prompt to fuse PLMs and Graphs for extracting task-specific and graph-aware 352 node representation in TAGs. G-Prompt have two-stage: (1) self-supervised train a graph adapter to 353 make PLMs graph-aware based TAGs, and (2) employing prompts with the trained graph adapter to 354 extract node representation from TAGs. Experiments with different shot settings using three datasets 355 demonstrate that the proposed model can effectively capture both text and graph information, resulting 356 in improved performance for few-shot learning. In zero-shot learning, our model achieves comparable 357 performance with supervised baselines and has huge potential for future work. Furthermore, our 358 model provides useful interpretations, which is essential for understanding the tasks and TAGs. 359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. arXiv preprint arXiv:2004.12651 , 2020.

Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic, and Inderjit S Dhillon. Node feature extraction by self-supervised multi-scale neighborhood prediction. arXiv preprint arXiv:2111.00064 , 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Hande Dong, Jiawei Chen, Fuli Feng, Xiangnan He, Shuxian Bi, Zhaolin Ding, and Peng Cui. On the equivalence of decoupled graph convolution network and label propagation. In Proceedings of the Web Conference 2021 , pages 3651-3662, 2021.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 , 2021.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332 , 2021.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced 384 bert with disentangled attention. arXiv preprint arXiv:2006.03654 , 2020. 385

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, 386 Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv 387 preprint arXiv:2106.09685 , 2021. 388

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele 389 Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. 390 Advances in neural information processing systems , 33:22118-22133, 2020. 391

Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, 392 Haizhen Huang, Denvy Deng, and Qi Zhang. Promptbert: Improving bert sentence embeddings 393 with prompts. arXiv preprint arXiv:2201.04337 , 2022. 394

Seungbae Kim, Jyun-Yu Jiang, Masaki Nakada, Jinyoung Han, and Wei Wang. Multimodal 395 post attentive profiling for influencer marketing. In Proceedings of The Web Conference 2020 , 396 pages 2878-2884, 2020. 397

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint 398 arXiv:1611.07308 , 2016. 399

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient 400 prompt tuning. arXiv preprint arXiv:2104.08691 , 2021. 401

Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural 402 networks with 1000 layers. In International conference on machine learning , pages 6437-6449. 403 PMLR, 2021. 404

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and 405 Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context 406 learning. Advances in Neural Information Processing Systems , 35:1950-1965, 2022. 407

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt 408 understands, too. arXiv preprint arXiv:2103.10385 , 2021. 409

410

411

412

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786 , 2021.

Bhaye Malhotra and Dinesh Kumar Vishwakarma. Classification of propagation path and tweets for rumor detection using graphical convolutional networks and transformer based encodings. In 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM) , pages 183-190. IEEE, 2020.

Costas Mavromatis, Vassilis N Ioannidis, Shen Wang, Da Zheng, Soji Adeshina, Jun Ma, Han Zhao, Christos Faloutsos, and George Karypis. Train your own gnn teacher: Graph-aware distillation on textual graphs. arXiv preprint arXiv:2304.10668 , 2023.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066 , 2019.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. arXiv preprint arXiv:1908.10084 , 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

Yiping Yang and Xiaohui Cui. Bert-enhanced text graph neural network for classification. Entropy , 23(11):1536, 2021.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems , 33:5812-5823, 2020.

Yazhou Zhang, Dan Ma, Prayag Tiwari, Chen Zhang, Mehedi Masud, Mohammad Shorfuzzaman, and Dawei Song. Stance level sarcasm detection with bert and stance-centered graph attention networks. ACM Transactions on Internet Technology (TOIT) , 2022.

Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. Learning on large-scale text-attributed graphs via variational inference. arXiv preprint arXiv:2210.14709 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of representation learning on text-attributed graphs (TAGs), which has gained significant attention in recent years. The primary focus of this research is on few-shot node classification. Existing two-stage methods have been unsuccessful in effectively capturing the complex relationship between graph structure and textual features, as well as incorporating downstream task information. To address these limitations, the authors propose a novel approach called G-Prompt, which aims to simultaneously capture the graph topology and downstream task information. \n\nG-Prompt achieves this by enhancing the pretrained language model (PLM) with graph awareness. At the last layer of the PLM, a graph adapter is introduced, allowing the model to become aware of the graph structure. This graph adapter can be combined with task-specific prompts, enabling the generation of task-specific and graph-aware node representations.\n\nSTRENGTHS:\n- Clear motivation of the method\n- Well-written summary of existing work (taxonomy, pros and cons)\n- Good results in few-shot node classification\n- Valuable insights into the utility of graph information and prompts for both LMs and GNN based methods\n\nWEAKNESSES:\n- **Some tables and figures are not self-contained, such as Figure 1 and Table 2.** The caption is too concise for the readers to follow. For example, in figure 1, the authors are expected to summarize the proposed framework in one or two sentences to aid comprehension; For table 2, the meaning of the bold and underline should be explicitly specified to ensure clarity and understanding.\n- **Background, method, and implementation of few-shot learning is missing.** As few-shot learning is less common in the graph community compared to supervised learning, it is essential to provide additional explanations and context for unfamiliar readers. \nMoreover, when comparing to some existing approaches, such as GIANT, which were originally designed for supervised learning, it should explicitly discuss how these methods can be adapted or extended for few-shot learning scenarios. Also, some compared methods, such as GIANT, were intended for supervised learning. How to adopt them for the few-shot learning is not straight-forward and should be introduced.\n- **The efficiency of the proposed method is stated but not experimentally proven.**\nIn section 3.3, the authors claim that training the graph adapters can be achieved with very few computing resources. However, to support this claim, it is necessary to provide a comparison with the compared methods (w.r.t. training parameters, memory usage, training speed, and training time, etc). This empirical evidence would provide stronger support for the efficiency of the proposed method. Furthermore, it is worth noting that the training epoch for the graph adapter is 50 for ogbn-arxiv, while typically fine-tuning it for 3-5 epochs is sufficient[1]. Therefore, it raises concerns whether the additional training epoch may negate the benefits introduced by training only the graph adapter.\n\n\n\n**Reference:**\n[1] Learning on Large-scale Text-attributed Graphs via Variational Inference, ICLR 2023.\n\nQUESTIONS:\n- According to section 3.4, the node features are fed into subsequent GNNs but in Figure 1, the GNN encoder is not part of the framework, which is a little bit confusing.\n- The dimention of the node features need to reduced for practical usage (section 3.4). And it is heuristic to select the optimal M. I am wondering how sensitive the graph prompter is to the value of M, and if there is any discussion on the selection of the optimal M.\n\nLIMITATIONS:\n- The proposed framework is not directly compatible with large language models (LLMs). Nowadays, when people refer to prompt-based methods, they mainly focus on utilizing large language models such as GPT. These models are known for their powerful text modeling and reasoning abilities, which have the potential for performing few-shot learning on text-attributed graphs (TAGs) [2]. However, the framework proposed in this paper is not directly compatible with LLMs, as G-Prompt requires access to LLMs' latent embeddings or logits, which are not provided by models like GPT-3.5 and GPT-4.\n\n\n**Reference:**\n\n[2] Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT. arXiv preprint arXiv:2304.11116 (2023).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors present a new framework that combines the benefit of Graph models with the Large Language Models. The authors argue that with the current modeling paradigm, the LLMs are trained in a downstream task agnostic fashion although using prompts they could be fine-tuned for specific tasks. Graph networks provide a principled approach of inferencing over structured data (such as over Text-attribute graphs), however there is currently no good mechanism to combine graph inference with LLMs and tune the output from LLMs based on underlying structured data. To alleviate this problem, authors propose G-prompt - a new technique that allows them to incorporate graph structure into LLM fine-tuning. To do so, they fine-tune the last layer of LLM by utilising the graph structure information using the same pooling approach as used in a graph model. They show that their mechanism performs better than vanilla LLM training. They further show that their method is better than SOTA methods for few-shot node classification.\n\nSTRENGTHS:\n1. In this paper authors present a novel technique to combine LLM with graph models.\n2. The proposed technique has wider application potential as a number of real world applications rely on structured data.\n3. The method beats the current SOTA techniques for few-shot node classification.\n4. They achieve comparable to supervised techniques for zero-shot learning, which is impressive.\n\nWEAKNESSES:\n1. The proposed technique certainly combined the best of sequence modeling using LLM and structured data modeling using graph networks. It would be interesting to try out a simple technique of flattening the graph data into a sequence and training an LLM with it. \n2. It would be good to discuss the computation and efficiency tradeoffs with different experiments.\n\nQUESTIONS:\n1. Have you tried an experiment where you flatten the graph (for instance by appending tokens with some positional information) and fed it into an LLM. How does it compare with your proposed G-prompt work?\n\nLIMITATIONS:\n1. It would be interesting to see how this method performs when applied to large visual models (LVMs). Do we see similar trends?\n2. Other than classification, could this technique be used for other semantic understanding tasks such as similarity matching?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

comments: SUMMARY:
The paper feeds manually designed prompts to LLMs to get task-specific text features, instead of BERT-based fixed features. Then a GNN is applied on top of node features for node classification. Few-shot node classification on 3 datasets are conducted for evaluation.

STRENGTHS:
1.The motivation is clear and reasonable.

2.Results on few-shot node classification outperform previous methods.

WEAKNESSES:
1.The prompts need to be manually designed, which significantly harms the novelty of this work.  It would be more interesting if the prompts can be automatically and jointly learned with the graph part.

2.The writing need to get improved. For example, Figure 1 failed to help me understand the model design of this work. BTW, the word “random” in this figure is misspelled.

3.Experiments are conducted on a single task (few-shot node classification).

4.Related work is not complete. There have been some work focusing on graph+prompt [1,2].

[1]Gppt: Graph pre-training and prompt tuning to generalize graph neural networks

[2]Graphprompt: Unifying pre-training and downstream tasks for graph neural networks

QUESTIONS:
Refer to the weaknesses

LIMITATIONS:
No. Can this work be adapted to link prediction or graph-level tasks?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents G-Prompt, a novel framework designed to model Text-attributed Graphs (TAGs) more efficiently. G-Prompt addresses the existing limitations of current methods by combining a graph adapter with task-specific prompts to extract node features, thereby integrating information from both the graph structure and downstream tasks. The graph adapter makes pre-trained language models (PLMs) aware of the graph structure, and the task-specific prompts provide task-related interpretations. Experimental results reveal that G-Prompt outperforms existing methods in few-shot learning and demonstrates robust performance in zero-shot settings. The generated node representations also exhibit high interpretability concerning task performance, indicating the framework's effectiveness in harnessing both graph and task-specific information.\n\nSTRENGTHS:\n1. The paper introduces a framework, G-Prompt, which effectively integrates graph and task-specific information for node feature extraction in text-attributed graphs (TAGs), which addresses a significant gap in existing methods.\n2. G-Prompt outperforms state-of-the-art methods on few-shot node classification and performs comparably with fully-supervised baselines in zero-shot settings.\n\nWEAKNESSES:\n1. While the G-Prompt framework is innovative, it builds heavily on existing concepts such as PLMs, GNNs, and the use of prompts. The novelty lies more in the combination and application of these techniques rather than in entirely new concepts.\n2. The experiments could have been strengthed by utilizing more advanced pre-trained language models, such as ALBERT, DeBERTa, etc., to demonstrate its wide applicability to other PLMs.\n\nQUESTIONS:\nMajor: Please address the weaknesses.\n\nMinor: There are several typos and stylistic issues. Please carefully revise your manuscript.\n\n1. Table 1: what is \"# Eeges\"? might be 'edges'?\n2. Figure 1: Romdom mask -> Random mask\n3. line 85: fuction -> function\n4. line 176: it's -> its\n\nLIMITATIONS:\nThe authors have not explicitly discussed the limitations or potential negative societal impacts of their work in the sections of the paper that were processed. While the authors demonstrate the effectiveness of G-Prompt on few-shot and zero-shot learning scenarios, it's unclear how well the framework generalizes to other types of tasks or datasets. The authors could discuss this potential limitation and provide insights to address it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new framework called G-Prompt for analyzing text-attributed graphs, which are commonly found in real-world networks. The existing methods for analyzing these graphs have limitations in improving performance when there is limited training data. G-Prompt addresses this issue by combining a graph adapter and task-specific prompts to extract better features from the graph. The experimental results show that G-Prompt outperforms existing methods in classifying nodes with limited training data. It also provides more understandable results and performs comparably to fully-supervised approaches in scenarios where no training data is available.\n\nSTRENGTHS:\n1. The paper addresses an interesting problem setting of learning on text-attributed graphs (TAGs). This problem setting has practical implications and contributes to advancing the understanding of graph-based machine learning.\n2. The proposed G-Prompt framework demonstrates good performance compared to baseline methods. It outperforms existing state-of-the-art approaches in node classification tasks, particularly in few-shot learning scenarios.\n\nWEAKNESSES:\n1. The proposed method lacks sufficient novelty as it combines existing techniques of graph adapters and prompt-based embedding learning, resulting in limited technical innovation.\n2. The ablation study of different modules in the proposed framework is lacking. The authors do not sufficiently analyze the individual contributions and impact of each component. In other words, how do the prompting and the graph adapter raise the performance individually?\n3. The description of baselines is unclear as there are no citations or explanations provided. It is unclear what the baseline GAE and GIANT refer to, and this lack of clarity hinders the reader's understanding and evaluation of the proposed method.\n4. The paper contains several typographical errors, such as the misspelling of \"GIANT\" as \"GAINT\" in line 242 and the misspelling of \"random\" as \"rondom\" in Figure 1. These errors suggest a need for thorough proofreading.\n\nQUESTIONS:\n1. The authors are doing classification tasks on all three datasets, however, they apply different evaluation metrics to them (ACC for Arxiv and ROC for Instagram and Reddit). Is there a reason for doing so? It's better to have some discussion here.\n2. Could the authors further explain the motivation of the GraphAdapter? How it becomes context-friendly and prompting-friendly?\n\nLIMITATIONS:\nNo, didn't see any limitations addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}