An Empirical Study Towards Prompt-Tuning for Graph Contrastive Pre-Training in Recommendations
Authors: Haoran Yang
Xiangyu Zhao
Yicong Li
Hongxu Chen
Guandong Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: graph contrastive learning
prompt tuning
recommendation system
TL;DR: An empirical study focuses on the limitations of graph contrastive pre-training in recommendation tasks and leveraging the prompt-tuning method to address such limitations.

Abstract: Graph contrastive learning (GCL) has emerged as a potent technology for numerous graph learning tasks. It has been successfully applied to real-world recommender systems, where the contrastive loss and the downstream recommendation objectives are always combined to form the overall objective function. Such a strategy is inconsistent with the original GCL paradigm, where graph embeddings are pre-trained without involving downstream training objectives. In this paper, we innovatively propose a prompt-enhanced framework for GCL-based recommender systems, namely CPTPP, which can fully leverage the advantages of the original GCL protocol through prompt tuning. Specifically, we first summarise user profiles in graph recommender systems to automatically generate personalized user prompts. These prompts will then be combined with pre-trained user embeddings to conduct prompt-tuning in downstream tasks, thereby narrowing the distinct targets between pre-training and downstream tasks. Extensive experiments on three benchmark datasets validate the effectiveness of CPTPP against state-of-the-art baselines. A further visualization experiment demonstrates that user embeddings generated by CPTPP have a more uniform distribution, indicating a better capacity to model the diversity of user preferences.
The implementation code is available online to ease reproducibility: https://anonymous.4open.science/r/CPTPP-F8F4

An Empirical Study Towards Prompt-Tuning for Graph Contrastive Pre-Trainin in Recommendations
Haoran Yang
Xiangyu Zhao"
University of Technology Sydney haoran, yang- -estng-essudett.... uts. edu.
City University of Hong Kong xianzhao@cityu. edu hk
Yicong Li
Hongxu Chen University of Queensland hongxu. chen@ug edu. au
University of Technology Sydney yicong. 1i ii@suudent... uts edu. au
Guandong Xu" University of Technology Sydney guandong. xu@uts.edi edu.ar au
Abstract
Graph contrastive learning (GCL) has emerged as an effective technology for various graph learning tasks. It has been successfully applied in real-world rec- ommender systems, where the contrastive loss and downstream recommendation objectives are combined to form the overall objective function. However, this approach deviates from the original GCL paradigm, which pre-trains graph embed- dings without involving downstream training objectives. In this paper, we propose a novel framework called CPTPP, which enhances GCL.-based recommender systems by leveraging prompt tuning. This framework allows us to fully exploit the advan- tages of the original GCL protocol. Specifically, we first summarize user profiles graph recommender systems to automatically generate personalized user prompts. These prompts are then combined with pre-trained user embeddings for prompt tuning in downstream tasks. This helps bridge the gap between pre-training and downstream tasks. Our extensive experiments on three benchmark datasets confirm the effectivennss of CPTPP compared to srate-ofthhe-as art baselines. Additionally, a visualization experiment illustrates that user embeddings generated by CPTPP have more uniform distribution, indicating improved modeling capability for user preferences. The implementation code available onlind for reproducibility.
Introduction
Graph contrastive learning (GCL) has gained significant attention in the research community prominent self-supervised learning paradigm. Several recent studies have showcased the effectiveness of GCL in various general graph representation tasks and link prediction. Moreover, GCL has also demonstrated its applicability in real-world domains 29)] By introducing additional self-supervision signals, GCL provides recommender systems with a means to address the challenge of improving performance.
28 37
including node classification
such as recommender systems 27 32
Most recommendation methods based GCL typically combine contrastive loss with recommenda- tion objectives to optimize the model in an end-to-end manner. However, this training protocol does
Corresponding author.
https://github wwt///aaaaaaaooooo///ee.a
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

not align with the purpose of GCL, which is primarily designed for pre- pre-training graph representations without involving downstream task objectives 26] In this approach, GCL pre-trains embeddings that are then fine-tuned on specific tasks using downstream models. Incorporating both GCL. and recommendation objectives into the overall training objective can disrupt the embedding pre-training process and requires careful control of the weight placed on contrastive loss. Additionally, previous studies on GCL-based recommendation methods [27 [4] have shown that the weights of contrastive loss in the overall objective are significantly smaller compared to the weight on the recommendation objective. This is done to ensure desired performance on recommendation tasks. Therefore, based on these observations, simply combining contrastive loss with downstream recommendation objectives may not be effective for recommendation tasks.
The disparity between the pre- training objective and downstream tasks hinders the effective extraction of useful information from pre-trained embeddings by downstream models [12 26]. Consequently, researchers often opt for combining GCL. with recommendation objectives. However, it is important to note that GCL pre-training targets primarily assess the agreement of mutual information among graph elements, such nodes, edges, and sub-graphs. This differs from conventional graph learning tasks like node classification and link prediction. Consequently, the pre-training targets of GCL also significantly diverge from downstream recommendation objectives that involve interaction (link) prediction between users and items. Consequently, the reduction of such dissimilarities is essential to enhance the performance of GCL-based recommendation approaches.
In this paper, we present the CPTPP framework as an extension of recent advancements in prompt tuning for enhancing recommendation performance [2 33
utilizing user embeddings pre-trained by GCL. The technique of prompt tuning has emerged as a prominent method for fine-tuning pre-trained models. By constructing appropriate prompts for downstream learning modules, this approach effectively reformulates downstream tasks, thereby reducing disparities [2 15 [8].
By incorporating prompt-tuning, we can modify existing GCL-based recommendation methods to align with the original GCL protocol involving pre-training and fine-tuning. Previous endeavors have also explored the integration of prompt learning into conventional recommendation models [26] 3
Despite their advantages, applying the prompt mechanism directly to GCL-based recommendation methods is still difficult and not straightforward, i.e., how can generate personalized user prompts using only the user-item interaction graph without side information (c.g., age and occupation)? To address this issue, we summarise three methods produce different user profiles, including historical interaction records, adjacency matrix factorization, and high-order user relations, based on the user-item interaction graph for the personalized user prompt generation, which is applicable in situations devoid of side information. Comprehensive experiments conducted on three publicly available datasets illustrate the effectiveness of the proposed method with different types of prompts.
In summary, the contributions of this work are three-fold: (1) We propose a reformulation of existing GCL-based recommendation methods by incorporating the prompt tuning mechanism. This allows us to fully leverage the advantages of GCL during the pre- pre-training phase, rather than relying on the combination of contrastive loss with downstream objectives. (2) We introduce three user profiles derived from the user-item interaction graph as inputs for the prompt generator. By using these profiles, we are able to generate personalized prompts that enhance the quality of user embeddings in graph-based recommendation systems. (3) We conduct extensive experiments on three publicly available benchmark datasets to validate the effectiveness of our model. Through these experiments, we analyze the important components and hyper-parameters of our approach and also investigate the impact of different personalized prompts generated by our method.
2 Methodology
In this section, the proposed method, graph Contrastive Pre- Pre-Trainin with PromPt-tuning for recom- mendation (CPTPP), will be introduced to reveal the intuitions and the technical details.
2.1 Framework Overview
There are three modules in the proposed CPTPP method, as shown in Figure 1. (1) graph contrastive learning module, leveraging the advantages of GCL to the pre-train user and item embeddings, (2) personalized prompts generation module, applying prompt mechanism to generate personalized
2

---

Graph Contrastive Learning
Recommendation Model
Pre-Trainn ro-iTrined Item
Embeddings
User-ltem Interaction Graph
Pre Trained User Embeddings
User Embeddings
Augmented User-Ttem
with Prompts
Interaction Graphs
Fusion
Personalized Inputs Pool
Concat
Prompt
Personalized Prompts
Concatenate Operation
Personalized Prompts & User Embeddings
Generator
Figure 1: The overview of the proposed method CPTPP.
prompts for users, and (3) recommendation module, fusing the generated personalized prompts and pre-trained user embeddings conduct prompt-tuning for the downstream recommendation task.
2.2 Graph Contrastive Learning Module
In order to achieve optimal performance in downstream tasks, the selection of a suitable pre-training strategy is crucial for generating high-quality inputs for downstream modules. GCL has been demonstrated as powerful technique for graph pre-training [21 16 30 37 and has emerged as effective tool for leveraging self-supervised signals to enhance graph-based recommendation models [27
14 24 32) In the case of graph-basee based recommender systems, GCL represents viable option for pre-training embeddings. Furthermore, our work specifically focuses on reforming and improving existing GCL-based recommendation methods. Therefore, it is imperative that we formulate GCL module within our proposed method. Current GCL.-based recommendation methods [14] 24 27 32
have explored various graph augmentation techniques user-item interaction graphs in order to generate augmented graphs for GCL, enabling the extraction of informative semantics from the graph structures. Alternatively, some studies B][41 have designed context embeddings tailored for GCL in recommended settings. Although different approaches exist for constructing contrasting samples, they all share common backbone for the GCL training protocol.
Here, we give a formal description of the GCL training protocol. Let u, denote the target graph element (e.g., user node), u," represent the positive sample generated from u; (e.g., the neighbor node of the target node), and U {u, "'. Ug,n be the set of contrasting samples of u; (e.g., non-neighbour nodes of the target node). Considering the settings of the recommendation task, we use G' to represent the overall user-item graph, and all the target, positive sample, and contrasting samples are within graph G. To acquire embeddings of these graph elements, we adopt f(+) as the graph encoder to process them, and the target embedding is denoted by u, fu; G). ut = flu G) is the embedding of the positive sample, and {u,or , u are the list of embeddings of the negative contrasting samples. Following the settings of InfoNCE 20), the self-supervised learning objective can formulated as follows:
Coontra log exp(sim(u; )/7))
En exp(sim(u, TTT

---

where r is the temperature syper-parameter and sim(- -) is the similarity metric. In existing research works [24 27 32 31 14. researchers usually combine the aforementioned contrastive learning loss function with the recommendation objectives to formulate an overall objective function to train the model an end-to-end manner:
Loverall
Lree + Lecontra
where A a hyper-parameter that controls the weight of the contrastive learning objective. However, as mentioned in Section [ the proposed CPTPP adopts a pre-train, prompt, fine-tune manner to train the model and treats GCL as a pre-training task instead combining the contrastive loss with recommendation objectives. To leverage recent research progress in GCL, we can adopt various GCL learning methods tailored for the recommendation task here, like NCL [14], SGL [24], and SimGCL [32,, to obtain high-quality user and item embeddings. Then, the pre- sre-trined user and item embeddings will be processed by the prompt mechanism the following.
2.3 Prompts Generation Module
Following the pre-training phase, our method, named CPTPP, incorporates a personalized prompts generation module utilize the pre-trained user and item embeddings effectively. The primary objective of this module is to address the limitations present in existing prompt and recommendation research. Prior studies [88 [5] 26 have highlighted the triviality and sesource-intensii nature of hard prompt design, making it impractical for real-world scenarios. Additionally, we observe that most current approaches rely on side information (e.g., user descriptions) to generate prompts, and there lacks a specific paradigm for prompting in graph-based recommendation scenarios. To overcome these limitations, we propose the integration of a prompt generator [26] that generates personalized prompts tailored specifically for graph- -ased recommendation contexts.
2.3.1 Personalized Prompt Generator
The main scope of the generated prompts lies in narrowing the gap between the pre-training targets and the downstream objectives to utilize the pre-trained models or embeddings better. Some research works designed hard prompts tailored for recommendation tasks converting recommendation tasks into NLP tasks [3], which unifies multiple recommendation tasks in a single framework. For example, a convention recommendation task can converted to a sentence, 'User 123 will purchase item [id_token]'. Then, NLP techniques will be applied to predict the token. However, PPR (26) argued that such an NLPPstyle hard prompt designing method has two major limitations: (i) is difficult to apply NLP techniques predict the designated tokens since these tokens could be user ID, item ID, or ratings, which lack meaningful semantics. (ii) The designed hard prompts are universal and cannot be customized for various users items.
To address the challenges, we adopt method to construct personalized prompts from user profiles in a soft prompt automatic generation manner [2] 15 26 Let x² E Rdx1 denote the profile of user i. We, then, concatenate all the users' profiles to form the user profile matrix X [x] xx,," ... , x
...
Rdx= This matrix will be fed into a two-layer perceptron f(-) acquire personalized prompts for each user P = -p" Ph E RPx= as follows:
pa= f(X") = We a(W, x +
(3)
where p is the prompt size, W, R.bxd and W2 Rpxh trainable weights, b, E Axxx and by E RRdxn are biases, and a<-) is the activation function. d and represent the dimensions of the pre-trained embeddings and hidden dimensions, respectively. The generated prompts will be concatenated with the pre-trained user embeddings in pre-fixed manner [12] and tuned by the downstream objectives Specifically, let U Upre_tree pre_trein E Bdxn denote pre- pre-trained user embeddings. Then, we have the inputs from the user side for the recommendation module:
the recommendation module fulfill the process of prompt-tuning,
p
Cconcat
E (x+dd))
(4)
U
pre_train
2.3.2 Personalized Inputs for Prompt Generation
The quality of the generated prompts depends on the personalized inputs for the generator. Current research on prompt learning for recommendation mainly focuses utilizing existing user features

---

(e.g., age, gender, and occupation) and historical interaction records as the inputs to generate person- alized prompts 266 II 25]. However, these methods are designed for conventional and sequential recommendations, which are not entirely aligned with graph recommendations, It is necessary to summarise and explore how to generate personalized prompts from the perspective of the graph recommendation system. In this section, we summarise three types of inputs for the generator to generate personalized prompts for the graph-based recommendation: historical interaction records, adjacency matrix factorization, and high-order user relations.
Historical Interaction Records. It is a common and widely-used method illustrate users' features or preferences via aggregating his/her historical interaction records, which is feasible in various scenarios in recommendation systems. Let I {ik,1 ik.2, ik.m } denote the item set which are purchased by user k. We use ig. I represent the embedding of the j-th item in user k's purchase history. Then, the profile of user i can be acquired by aggregating embeddings of those items purchased by the user
x Aggregation (ix,1, is.2, ik.m),
(5)
where Aggregation(+) is the aggregation function to read out the user's profile.
The concatenation of all the user profiles can form the matrix X " to be processed by the personalized prompt generator. Let A Ix* denote the adjacency matrix for the recommendation system, which contains n users and items. we have the pre-trained item embeddings Ipre_train =
[i, ...,.,,,i, , E Rdxq, then, we have:
X x (A I (pre_train)
(6)
Adjacency Matrix Factorization. The adjacency matrix is an effective tool to demonstrate the user- item relations the recommendation system. However, the adjacency matrix usually suffers from sparsity problems and thus cannot be smoothly applied in many real-world recommendation scenarios. To address this problem, researchers proposed several matrix factorizations (MF) methods [T9] to decompose the adjacency matrix to obtain two matrices, U and V, denoting the latent embeddings for users and items, which are much denser than the adjacency matrix itself. The process of MF can be formulated as follows:
arg min U.V
Aiij
Aig);
f=
where Ái E -,VJ After the MF process, we have the latent matrix of users, U E IRaxd, serving as the user profile matrix X after transposed X = U and can be fed into personalized prompt generator to produce personalized prompts P for each user. Specifically, we set the size of latent embeddings d, the same the size of pre-trained embeddings.
High-Order User Relations. Learning informative embeddings from 1 1-hop user-item interaction graph is challenging when there is no side information. To address this limitation, we propose to leverage high-order user relations enrich the learned embeddings via constructing 2-ego graphs for each user node to find the links between the other users and itself [22). Then, we fuse the target user's purchase history and high-order neighbor embeddings to represent the target user profile.
We first construct the high-order connectivity matrix to achieve the goal. Let A
À E
A
R ((++)))((++)) denote the high-order connectivity matrix, where A AT
E (
0
recording all the users and items to which user or item node is connected. Then, we build the matrix E = UU pre_trainn Ip Ipre_trein]" E x to store pre-trained embeddings. Next, we can acquire matrix Q Ruxd which are the users' personalized profiles about high-order user relations, via:
Q M
E..EERRR ((+g))xd E(ttg)))
(8)
where M E Ra denoting the high-order item relations and being omited after Q is extracted. Then, matrix transpose operation is required to obtain the user profile matrix X Q
5

---

2.4 Recommendation Module
After the pre-training and the personalized prompts generation phase, recommendation module required so that we can verify whether the prompt-tuning module rectifies the pre-trained embeddings by GCL and makes them adapted to the downstream recommendation tasks better. In this module, we take the inner product of user and item embeddings the predicted score for the recommendation. Bayesian Personalized Ranking (BPR) [17] is adopted as the training objective to tune the pre- trained embeddings based the predicted scores. The motivation for formulating such a simple recommendation module is to avoid the performance gain brought by the delicate designs of those proposed method.
advanced recommendation models, which could affect the observations
Algorithm 1: CPTPP algorithm
Input: User embedding table Ugi Item embedding table Ig: User-item interaction graph adjacency matrix A; Graph contrastive learning model f()); User profile X"; Prompt generator g(-); Multi-layer perceptron MLP (-); Pre-train epoch i; Prompt-tune epoch j. Output: User and item embedding tables U and Igg-
Pre-train phase:
Initialize Ug, Ig; L'UUF, IF
UE Lg;
3 count = 0;
while count i do
// Update user and item embedding tables.
U L f(U Ri I A);
count count + 1; end Prompt-tune phase:
U < U g Ig Ig:
10 count 0;
while counl < j do
// Personal ized prompt generation.
12
Dd gX"):
// Concatenate fusion.
U MLP([P"; ) Raxd, Optimise L =ieu free + X|
15
Update U 1;;
16
count count + 1;
17 end
18
return U F
2.4.1 Prompts and Pre-Trained Embeddings Fusion
We concatenate the generated personalized prompts and the pre-trained user embeddings in the previous step and have U Uconcat E (p-ddd whose dimensionality is different from the pre-trained embeddings Apre_traie
Rga Hence, we need first to fuse the personalized prompts and the pre-trained user embeddings for the recommendation objective training. Specifically, we adopt multi-layer perceptron (MLP) 9(-) as the mapping function that is g (x++d)) - R Then, we can have dimensionality-reduue personalized prompts. After that, we can apply the inner product to predict how likely the user i would interact with the item j by Di.j uf ij. where u the i-th row of U*.
user representations U g( U concaat) gaxd enhanced by the
2.4.2 Training Objective for Recommendation Task
For simplicity and fair comparison, we adopt BPR loss as the training objective for the recom- mendation task. For each user i, we have;
C Σ Σ
- log (0i.s D1.g-).
(9)
I+

---

However, it is unaffordable to consider all the unobserved interactions of the user i. Therefore we sample several negative items NV", where [V22 << [ in practice.
Moreover, we introduce 1.2-norm into the training objective to regularize the parameters in the model address the overfitting problem and improve generalization ability. Therefore, the overall objective function can be formulated as:
E + 2l|l1k2 iEM
(10)
2.5 Summary
After the training process ends, the model can be used conduct inference. For the inference phase, we do not conduct pre-train and prompt-tune again. What we need to do is to extract target user and item embeddings from the trained embedding tables. Then, calculate their inner product predict the probability that the user will interact with the item in the future.
The complete training procedure of CPTPP is illustrated by Algorithm I We first initialize the user and item embedding tables (line 2). Then, we apply a GCL model to conducting embedding pre-training (line 4 7). Next, we step into the prompt-tuning phase and assign the pre-trained embeddings to U and I (line 9). Following, we input the user profile to the prompt generator to produce the personalized prompts (line 12) and combine them with U (line 13). Finally, use U and I to calculate the loss and update them accordingly (line 14 15). The update procedure will repeat until the termination condition is achieved (line 11 17).
3 Experiment
To verify the effectiveness of the proposed method, CPTPP, in this paper, we conduct extensive experiments and demonstrate the results with insightful analysis in this section.
Table 1: Dataset Statistics
Dataset
#Users
#ltems
#interactions
Density
Douban
2,848
39,586
894,887
0.794%
ML-IM
6,040
3,900
1,000,209
4.246%
Gowalla
29,858
40,981
1,027,370
0.084%
3.1 Experimental Setup
This section introduces the experimental settings, including datasets and baselines we used, perfor- mance metrics, and syper-parameter settings for CPTPP. More details about how get access the datasets and implementation details are listed in Appendix A
Datasets To verify the performance of CPTPP in the recommendation task, select three popular datasets: Douban [4, MovieLens- 1M [5 and Gowalla [33]. The detailed statistics about the three datasets listed in Table [ For each dataset, randomly select 80% of historical user-item interactions as the training set, and the rest 20% records will serve as the testing set. Following the settings widely adopted by the research community [2 6 we treat each user-item interaction record as a positive instance and conduct negative sampling to couple it with a negative instance, which is an unobserved user-item interaction the dataset.
Baselines We select several baselines for comparison experiments: BPR-MF 9), BUIR [T), SelfCF [36. NCL [4, and SimGCL 32] For CPTPP, have three variations, which are CPTPP-H. CPTPP-M, and CPTPP-R, respectively. -H takes historical interaction records for personalized prompt generation. -M indicates that take adjacency matrix factorization for personalized prompts generation. Furthermore, -R takes high-order user relations for the personalized prompt generation.
Metrics To evaluate the quality of top-K recommendation, we adopt three popular metrics, which are Hit Ratio@ K, Precision @ K, and NDCG@ K, respectively. In our settings, the value of K is set to 5 and 20. Following the evaluation protocol in [4 32] we take the full ranking strategy [35].

---

Hyper-parameter Seftings To ensure reproducibility, we disclose the comprehensive hyper- parameter settings for implementing our proposed CPTPP in the source codes and Appendix A.3
3.2
Experiment Results
We conduct experiments and provide analysis in this section. More supplementary experiment results, including comparison, hyper-parameten and ablation study, are revealed in Appendix B
Table 2: The experiment results of comparison studies. The figures in boldface indicate the best performance achieved by one of the three versions of CPTPP, and the figures underlined indicate the best performance among all the baselines.
Methods
Datasets
Metrics
BPR-MF
BUIR
SeFCF
NCL
SimGCL
CPTPP-H
CPTPP-M
CPTPP-R
Hit Ratio@5
0.0134
0.0156
0.0161
0.0161
0.0161
0.0164
0.0165*
0.0164
Hit Ratio
0.0446
0.0492
0.0502
0.0507
0.0489
0.0521
0.0528*
0.0523
Precision@
0.1812
0.2113
0.2185
0.2187
0,2182
0,2221
0.2235*
0.2224
Douban
Precision
0.1512
0.1667
0.1699
0.1717
0.1657
0.1766
0.1790
0.1772
NDCG@5
0.1904
0.2209
0.2264
0.2313
0.2370
0.2359
0.2378
0.2355
NDCG@ 20
0.1749
0.2019
0.2058
0.1958
0.2020
0,2065
0.2098
0,2070
Hit Ratio 5
0.0469
0.0617
0.0624
0.0655
0.0631
0,0676
0.0674
0.0672
Hit Ratio
0.1454
0.1519
0.1643
0.1796
0.1698
0.1851
0.1861
0.1845
Precision
0.1800
0.2368
0,2396
0,2513
0.2420
0.2592
0,2585
0.2577
ML-
Precision@20
0.1395
0.1457
0.1576
0.1723
0.1629
0.1776
0.1785
0.1770
NDCG@5
0.1968
0.2722
0.2689
0,2818
0.2767
0.2919
0,2895
0.2878
NDCG@20 20
0.2103
0.2367
0,2508
0.2683
0,2670
0,2781
0,2782
0.2756
Hit Ratio
0.0429
0.0479
0.0497
0.0488
0.0513
0.0518
0.0512
0.0519
Hit Ratio
0.1039
0.0993
0.1042
0.1040
0.1065
0.1115
0.1103
0.1120
Gowalla
Precision
0.0624
0.0698
0.0723
0.0711
0.0746
0.0754
0.0745
0.0755
Precision@@20
0.0378
0.0361
0.0379
0.0378
0.0387
0.0406
0.0401
0.0407
NDCG@5
0.0770
0.0911
0.0939
0.0894
0.0963
0.0963
0.0953
0.0961
NDCG@20
0.0939
0.0990
0.1036
0.1005
0.1126
0.1092
0.1083
0.1092
* indicates that CPTPP outperforms the best baseline significantly (i.c., two-sided t-test with 0.05).
3.2.1 Overall Comparison Studies
Table Sshows the comparison results among all the baselines and different versions of the proposed methods. (i) We can first observe that the traditional method BPR-MF is outperformed by all the other methods as they utilize contrastive learning to introduce extra unsupervised training signals. (ii) Among all the baselines, GCL-based recommendation methods, including NCL and SimGCL, significantly and consistently outperform those self-supervised recommendation methods without graph learning module equipped, BUIR and SclfCF. It is because those GCL-based methods adopt graph neural networks, leveraging the sophisticated structure semantics in user-item interaction graphs to enrich learned user embeddings and item embeddings. (iii) But we notice that SimGCL only outperforms NCL on dataset Gowalla, which has much larger scale than the others, probably because SimGCL adopts simplified GCL method that relieves the model overfitting problem on a large-scale dataset. It is the potential reason NCL outperforms SimGCL on smaller datasets, as the simplified GCL method may not provide sufficient self superviedd training signals. (iv) Though the proposed CPTPP solely adopts BPR loss, which is significantly different from the pre-training procedure, for the recommendation task training, we utilize the prompt learning mechanism to better adapt the embeddings pre-trained by the GCL method to the downstream task, expecting to improve the recommendation performance. According to the experiment results, all versions our proposed method achieve competitive results. Such results reflect prompt-tuning's effectiveness in narrowing the gap between the pre-train objective and the downstream tasks.
To further evaluate the performance of the GCL-based recommendation methods, we visualize the produced user embeddings produced by t-SNE and Gaussian kernel density estimation (KDE). We can see that CPTPP has a more uniform distribution of the produced user embeddings, illustrated by the uniformity of the color maps, especially on dataset MI- 1M and Gowalla. As suggested by Z. Lin el al. [14], the more uniform the embedding distribution is. the more powerful the capability model the diverse preferences of users the produced embeddings will have, which reflects the superiority of CPTPP compared to the baselines. The visualizations and analysis are listed in Appendix B.1
8

---

40
(a) CPTPP- Douban
(b) CPTPP- MLL-IM
(c) CPTPP- -Gowalaa
Figure 2: The visualizations of the user embeddings generated by the proposed method.
3.2.2 Hyper-Parameter Studies
To investigate the properties of our proposed CPTPP method, we conduct hyper- parameer studies on an important term, the dimension size of the personalized prompt. By fixing all the other hyper- parameters, we comprehensively examine the performance of three versions of the proposed CPTPP on all the datasets with different prompt sizes. Specifically, the size of the personalized prompt selected from {8. 16,32, 64, 128, 256]. We choose two metrics, Precision @5 and NDCG@5, to demonstrate CPTPP's performance variations with regard to different prompt sizes. All the experiment results are shown in Figure 3 and Figure in Appendix [B.2] (i) The first finding we can observe is that, in most cases, CPTPP has the best performance when the prompt size is not larger than the dimensionality of user embeddings, i.e, 64, A potential reason is that the prompt is usually less informative than the pre-trained embeddings, so a sizeable prompt dimension would introduce too much noise to disturb and conceal the structural semantics contained in the pre-trained user embeddings. (ii) We also notice significant performance improvement when prompt size is 256 in several cases, such as CPTPP-M on dataset ML-IM and CPTPP-R on dataset Gowalla. Such outlier performance could be caused by random factors during the overall training process. However, they still fail to significantly outperform the CPTPP model, which has a much smaller prompt size. Therefore, small prompt size for prompt-tuning is a better option in practice as they achieve relatively better recommendation quality and higher efficiency.
0.400
025
0.200
0.290
0.000
0.100
Precikocc85
Precisona8
Preciton:855
0.255
NDDDR
NDC18
0.2425
NDC
0.098
0.201
0.230
NDCORS
0.076
0.00
0.260
0.074
0.094
0.0
0.2600
0.55
0.15
0.2825
0.092
0.10
0.290
00
0,2800
0.00
0.000
Prompe S1z0
Prompe St.0
Promge Stee
(a) CPTPP-H Douban
(b) CPTPP-H ML-1M
(c) CPTPP-H- Gowalla
Figure 3: The performance, demonstrated by Precision@5 and NDCG@5, of all variations of CPTPP
3.2.3 Ablation Studies
As we summarize three strategics to generate personalized prompts for users, we conduct the ablation study to explore the differences among these methods. Two ablation studies are conducted in this section to illustrate the performance of three variations of the CPTPP method. The first ablation study is about the overall evaluation of recommendation quality, whose analysis is listed below. The second one is about the embedding visualizations and the related analysis, illustrated in Appendix B.3
We notice that (i) CPTPP-M achieves the best performance on dataset Douban. Nevertheless, the performance of CPTPP-M degrades on dataset ML- and is the worst case on dataset Gowalla. Considering the number of users reflected in Table I we find that the performance of CPTPP-M drops as the dataset's number of users increases. So, CPTPP-M has good performance if the number of users in the dataset is relatively small. It may be because matrix factorization, as a naive method, cannot

---

fully reveal user preferences in complex user-item interaction graph with too many user nodes. (ii) CPTPP-R utilizes high-order relationships among users to enrich the generated personalized prompts for users. In such settings, the item information would also be aggregated due to the message-passae mechanism GNNs. Therefore, it achieves the best performance on the dataset Gowalla, having the most users and the most complex user-user relation among all the datasets. (iii) CPTPP-H has moderate performance. CPTPP-H adopts historical interaction records, formed by trainable item embeddings, to generate personalized prompts. Those trainable elements endow CPTPP-H H with more robust capability to represent user preferences than matrix factorization. also reasonable that CPTPP-R outperforms CPTPP-H as CPTPP-H lacks consideration of high-order user relations.
4 Related Work
4.1 GCL in Recommendation Systems
The GCL-based recommendation system is now a trending topic in the research community. Re- searchers leverage the advantages of GCL to improve current graph-based recommendation methods further, achieving satisfying performance. The research scope first lies in utilizing user-item interac- tion records to accommodate GCL techniques. For example, HMG-CR (27] innovatively proposes a concept called hyper-meta path. Then GCL. is applied to adaptively learn the behavioural patterns of users via contrasting different paths. NCL. [14] improves graph-based recommendation models via saghborrhod--nrichee contrastive learning, including semantic neighbors and structural neighbors (r-ego graphs). To better understand the role of GCL in the recommendation systems, SGL [24] conducts a comprehensive theoretical analysis, giving insights and achieving promising performance. Based on the findings of SGL,, SimGCL 32) is proposed to simplify GCL in recommendation via discarding complex augmentations, reducing the volume of GCL-based recommendation models, and performing competitive results in the experiments. However, current GCL-based recommendation methods combine the graph contrastive loss with the recommendation objectives to formulate an overall objective train the models, suffering from various limitations summarised in the previ- ous sections. Our method innovatively introduces the prompt mechanism to build a pre-train then prompt-tune paradigm for GCL-based recommender systems to address the limitations discussed.
4.2 Prompt-Tuning
Prompt-tuning is a novel and trending paradigm for pre-train models in the natural language process (NLP). The core idea of prompt-tuning is foroouullaee the downstream tasks, narrowing the huge gap between them and the pre-train objective [8]. There are two types of methods to achieve prompt-tuning (3) The first is manually designing or searching for proper discrete prompts (hard
prompts) 22
8 18 However, such fashion is trivial and resource-consuming as the search space is extremely large and expert knowledge is required (26 in some application scenarios. To address this limitation, another line of methods is proposed, focusing on generating continuous vector embeddings as the soft prompts [5]. The application of prompt-tunin
recommendation systems has been explored. For example, P5 [3] reforms the recommendation tasks to the NLP tasks and follows the hard-prompt fashion to perform recommendations. PPR (26), instead, takes soft-prompt and pre-fix strategy 12 to generate personalized prompts for users in the recommendation systems automatically. Nevertheless, graph learning and applications are now out of the scope of prompt-tuning research. Besides, most current prompt learning methods require side information to produce high-quality prompts, resulting in a limited comfort zone for its applications. Our CPTPP first adopts the prompt mechanism to the GCL-based recommendation area.
5 Conclusion
In this paper, we propose CPTPP method to adopt a prompt-tuning technique to reform and improve current GCL-based recommendation methods. To better accommodate prompt learning to graph recommendation scenarios, we summarise several graph-oriented user profiles to generate personal- ized user prompts to conduct prompt-tuning for downstream recommendation tasks. Comprehensive experiments have shown the effectiveness, superiority, and properties of the proposed CPTPP method. The future research directions about prompt-tuning GCL-based recommendation may be two-fold: how to (i) generate personalized prompts and (ii) integrate prompt-tuning strategy into GCL protocols.
10

---

Acknowledgments and Disclosure of Funding
This research work was supported by the Research Impact Fund (No. R1015-23), the Australian Research Council (ARC) under Grant Nos. DP220103717, LE220100078, LP170100891, and DP200101374 and was partially supported by APRC- - CityU New Research Initiatives (No. 9610565, Start-up Grant for New Faculty of City University of Hong Kong), CityU HKIDS Early Career Research Grant (No. 9360163), Hong Kong ITC Innovation and Technology Fund Midstream Research Programme for Universities Project (No. ITS/034/22MS), Hong Kong Environmental and Conservation Fund (No. 88/2022), SIRG CityU Strategic Interdisciplinary Research Grant (No.7020046, No. 7020074), Tencent (CCF-Tencent Open Fund, Tencent Rhino-Bird Focused Research Fund), Huawei (Huawei Innovation Research Program), Ant Group (CCF-Ant Research Fund, Ant Group Research Fund) and Kuaishou.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[2] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre- mr-itraened language models better few-shot learners. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 3816-3830, Online, August 2021. Association for Computational Linguistics.
[3] Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rip): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys '22, page 299-315, New York, NY, USA, 2022. Association for Computing Machinery.
[4] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few- shot learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8410-8423, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[5] Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4), dec 2015.
[6] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, YongDong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 339-648, New York, NY, USA, 2020. Association for Computing Machinery.
[7] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Weh, WWW 17, page 173-182, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee.
[8] Zhengbao Jiang, Frank F, Xu, Jun Araki, and Graham Neubig. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8:423-438, 07 2020.
[9] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37, 2009.

---

[10] Dongha Lee, SeongKu Kang, Hyunjun Ju, Chanyoung Park, and Hwanjo Yu. Bootstrapping user and item representations for ole-class collaborative filtering. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Reirieval, SIGIR "21, page 317-326, New York, NY, USA, 2021. Association for Computing Machinery.
[11] Lei Li, Yongfeng Zhang, and Li Chen. Personalized prompt learning for explainable recommen- dation. ACM Trans. Inf. Syst., 41(4), mar 2023.
[12] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join! Conference on Natural Language Processing (Volume I: Long Papers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics.
[13] Dawen Liang, Laurent Charlin, James McInerney, and David M. Blei. Modeling user exposure in recommendation. In Proceedings of the 25th International Conference on World Wide Web, WWW 16, page 951-961, Republic and Canton of Geneva, CHE, 2016. International World Wide Web Conferences Steering Committee.
[14] Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. Improving graph collaborative filtering with neighborhood-enriele contrastive learning. In Proceedings of the ACM Web Conference 2022, WWW '22, page 2320-2329, New York, NY, USA, 2022, Association for Computing Machinery.
[15] Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203-5212, Online, June 2021. Association for Computational Linguistics.
[16] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, page 1150-112 1160, New York, NY, USA, 2020. Association for Computing Machinery.
[17] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI '09, page 452-461, 461, Arlington, Virginia, USA, 2009. AUAI Press.
[18] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 4222-4235, Online, November 2020. Association for Computational Linguistics.
[19] Suvrit Sra and Inderjit Dhillon. Generalized nonnegative matrix approximations with bregman divergences. In Y. Weiss, B. Schölkopf, and Platt, editors, Advances in Neural Information Processing Systems, volume 18. MIT Press, 2005.
[20] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.
[21] Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference on Learning Representations, 2019.
[22] Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collabo- rative filtering. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR' 19, page 165-174, New York, NY, USA, 2019. Association for Computing Machinery.
[23] Yuhao Wang, Xiangyu Zhao, Bo Chen, Qidong Liu, Huifeng Guo, Huanshuo Liu, Yichao Wang, Rui Zhang, and Ruiming Tang. Plate: A prompt-enhanced paradigm for multi-ccenario recommendations, In Proceedings of the 46th International ACM SIGIR Conference Research
12

---

and Development in Information Retrieval, SIGIR '23, page 2023. Association for Computing Machinery.
New York, NY, USA,
[24] Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self-supervised graph learning for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 726-735, New York. NY, USA, 2021. Association for Computing Machinery.
[25] Yiqing Wu, Ruobing Xie, Yongchun Zhu, Fuzhen Zhuang, Ao Xiang, Xu Zhang, Leyu Lin, and Qing He. Selective fairness in recommendation via prompts. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '2, page 2657-2662, New York, NY, USA, 2022. Association for Computing Machinery.
[26] Yiqing Wu, Ruobing Xie, Yongchun Zhu, Fuzhen Zhuang, Xu Zhang, Leyu Lin, and Qing He. Personalized prompt for sequential recommendation, 2023.
[27] Haoran Yang, Hongxu Chen, Lin Li, Philip S. Yu, and Guandong Xu. Hyper meta- path contrastive learning for multi-behavior recommendation. In 2021 IEEE International Conference on Data Mining (ICDM), pages 787-796, 2021.
[28] Haoran Yang, Hongxu Chen, Shirui Pan, Lin Li, Philip S. Yu, and Guandong Xu. Dual space graph contrastive learning. In Proceedings of the ACM Web Conference 2022, WWW '22, page 1247, New York, NY, USA, 2022. Association for Computing Machinery.
[29] Haoran Yang, Xiangyu Zhao, Muyang Li, Hongxu Chen, and Guandong Xu. Mitigating the performance sacrifice in dp-satisfied federated settings through graph contrastive learning. Information Sciences, 648: I 19552, 2023.
[30] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5812-5822 Curran Associates, Inc., 2020.
[31] Junliang Yu, Hongzhi Yin, Jundong Li, Qinyong Wang, Nguyen Quoc Viet Hung, and Xi- angliang Zhang. Self-supervised multi-channel hypergraph convolutional network for social recommendation. In Proceedings of the Web Conference 2021, WWW '21, page 413-424, New York, NY, USA, 2021. Association for Computing Machinery.
[32] Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, and Quoc Viet Hung Nguyen. Are graph augmentations necessary? simple graph contrastive learning for recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22. page 1294-1303, New York, NY, USA, 2022. Association for Computing Machinery.
[33] Chi Zhang, Rui Chen, Xiangyu Zhao, Qilong Han, and Li Li. Denoising and prompt-tnning for multi-behavine recommendation. In Proceedings of the ACM Web Conference 2023, WWW "23, page 1355-1363, New York, NY, USA, 2023. Association for Computing Machinery.
[34] Guoshuai Zhao, Xueming Qian, and Xing Xie. User-service rating prediction by exploring social users' rating behaviors. IEEE Transactions on Multimediaa 18(3):996-50) 2016.
[35] Wayne Xin Zhao, Junhua Chen, Pengfei Wang, Qi Gu, and Ji-Rong Wen. Revisiting alternative experimental settings for evaluating top-n item recommendation algorithms. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM '20, page 2329-22222222 2332, New York, NY, USA, 2020. Association for Computing Machinery.
[36] Xin Zhou, Aixin Sun, Yong Liu, Jie Zhang, and Chunyan Miao. Selfef: A simple framework for self supervuper collaborative filtering. ACM Trans. Recomm. Syst., 1(2). jun 2023.
[37] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, WWW '21, page 2069-2080, New York, NY, USA, 2021. Association for Computing Machinery.
13

---

A
Reproducibility
This section provides supplementary details about our experimental settings for reproducibility.
A.1
Datasets
Three publicly available datasets are used in this work examine the performance of the proposed CPTPP. Here, we provide the links for downloading these datasets for readers to retrieve:
Douban: haabbannttt:::/aa..bbiiddd http:///w.aaaaouu.coooo///////or ptt.aa/ta.//ra//.a/aaa/ooco/o//n
MI-1M:
Gowalla:
A.2
Baselines
We select several baselines for comparison experiments, which are listed below:
BPR-MF adopts a matrix factorization framework to learn embeddings for users and items via optimizing the BPR loss function.
only uses positive user-item interactions to learn representations following a boot- strapped manner, consisting of an online encoder and target encoder.
BUIR 10]
SeIfCF [36 follows a similar strategy that BUIR adopts, which drops the momentum encoder to simplify the previously proposed method.
NCL [4] utilizes neighbor clustering to enhance GCL methods to acquire enhanced embeddings for users and items the recommendation system.
SimGCL 32 discusses the role of augmentations in GCL for recommendation tasks and proposes a simplified GCL method for recommendations.
A.3 Hyper-parameter Settings
Table 3: Summary of hyper-parameter settings of CPTPP.
Dataset ML-IM
Hyper-panann pareet-arrmettr
Notation
Douban
Gowalla
Hidden dimension size
d
64
64
64
Pre-train epoch
10
10
10
Prompt-tune epoch
100
100
100
Batch size
512
512
2048
Learning rate
0.003
0,001
0.001
Regularizer weight
X
0.0001
0.0001
0.0001
Number of GNN layers
2
Dropout rate Temperature parameter
0.1
0.2
0.2
0.2
[8. 16, 32, 64,
[8. 16, 32, 64.
[8, 16, 32, 64.
Prompt size
P
128, 256)
128, 256]
128. 256)
We list detailed hyper- paramete settings here for reproducibility. The dimensionality of the repre- sentation embeddings of users and items is set to 64, and the personalized prompt size is chosen from f8, 16, 32, 64, 128). For the pre-train phase, the maximum training epoch is 10. and for the prompt-tune stage, the training epoch is set to 100. The training batch size is 512 for the relatively smaller datasets, including Douban and ML- IM. For Gowalla, and À are set to - and - where À is the weight for the 12-norm term in the overall training objective. The default number layers of graph neural networks used in the models is set These settings are summarised in Table 3
set to 2048. The learning rate

---

45
(a) NCL -ouban
(b) SimGCL -Dobaan
(c) NCL ML-IM
-1
(d) SimGCL ML-1M
MIGGLL-ML-M
(c) NCL Gowalla
(f) SimGCL Gowalla
Figure 4: The visualization results of the user embeddings generated by baselines.
B Supplementary Experiment
In this section, several supplementary experiments are provided. Due to the page limit, the supple- mentary experiment results are listed and analyzed in the following instead of the main content.
0.200
0.665
0.550
0080
6.00
Prociionn NDCOWS
Precklns85
PacisknnS
NDCOHR5
DOTR
NDCORS
0.230
0.260
0.2000
p.14
2.09
0255
0.21%
02322
0012
0.21
Pronggt
Prorept Sioc
Photzt Sioce
(a) CPTPP-M Douban
(b) CPTPP-M ML-IM
(c) CPTPP-M Gowalla
0.24
025
0265
0250
0050
A.00
PicciiionS KpcoBs
Prechiun83
VDS Paxision8
NDOG-B5
NDOGes
0.22
0226
0076
6096
D258
0284
0074
6034
0.19
0.20
0245
256
0.200
0070
0.000
(d) CPTPP-R Douban
(c) CPTPP-R ML-IM
(f) CPTPP-R Gowalla
Figure 5: The performance, demonstrated by the metrics Precision @5 and NDCG@5, of CPTPP-M and CPTPP-R the selected datasets.
B.1 Supplementary Comparison Study
According to Figure 4 We can see that (i) embeddings learned by SimGCL fall into several hot areas on dataset M, and they are centralized in a small area on datasets Douban and Gowalla. (ii) NCL exhibits better performance as the distribution of the user embeddings expands to relatively larger area than that of SimGCL. Compared to our proposed method CPTPP, we can observe that CPTPP has more uniform distribution of the produced user embeddings, illustrated by the uniformity of the

---

color maps, especially on dataset ML- 1M and Gowalla. As suggested in [1], the more uniform the embedding distribution is, the more capability to model the diverse preferences users the method has, which reflects CPTPP's superiority.
B.2 Supplementary Hyper-Parameter Study
In Section 3.2.2 we solely illustrate the performance of CPTPP-H with different prompt sizes on the three datasets. We show the rest of the hyper-paramenn parameter study results in Figure 5 For easy reading, we list our findings here again: The first thing we can observe that, most cases, CPTPP has the best performance when the prompt size is not larger than the dimensionality of user embeddings. A potential reason that sizeable prompt dimensions would introduce more noise into pre-trained user embeddings, disturbing the structural semantics extracted from the user-item interaction graph by graph contrastive learning. (ii) We also notice significant performance improvement when prompt size is 256 in several cases, such as CPTPP-M on dataset ML-1M and CPTPP-R on dataset Gowalla. However, they still fail to significantly outperform the CPTPP model, which has a much smaller prompt size. Therefore, small prompt size for prompt-tuning is a better option in practice as they achieve relatively good recommendation quality and higher efficiency.
64
35-
12
20
-01
22
(a) CPTPP-H Douban
(b) CPTPP-M Douban
(c) CPTPP-R Douban
14
02
45
25
25
(d) CPTPP-H MI-IM (c) CPTPP-M ML.-M (f) CPTPP-R ML-IM
(g) CPTPP-H Gowalla (h) CPTPP-M Gowalla
(i) CPTPP-R Gowalla
Figure 6: The visualizations of the user embeddings generated by different versions of CPTPP.
B.3 Supplementary Ablation study
The impacts of different personalized prompts on CPTPP investigated. We visualize the user embeddings produced by all three variations of the proposed CPTPP as shown in Figure ( We can observe that both CPTPP-H and CPTPP-R have more uniform distribution, especially on datasets Douban and ML-IM. Such an observation indicates that personalized prompts generated from trainable user profiles can produce user embeddings that have more uniform distributions to demonstrate diverse user preferences better.


Reviews:
[{"review_id": "cve9BvgvBB", "reviewer": "Reviewer_PJh3", "summary": "This paper proposes a prompt-enhanced framework for GCL-based recommender system, called CPTPP. CPTPP reforms the existing GCL-based recommendation methods with the prompt tuning mechanism to fully exploit the advantages of GCL in the pre-training phase instead of combining the contrastive loss with downstream objectives. The authors summarize three user profiles derived from the user-item interaction graph as the inputs, without requiring extra side information, for the prompt generator. Extensive experiments on real-world datasets have demonstrated the effectiveness.", "questions": "1. What are the backbone GCLs of these models in the experiments? Do the generation modules perform differently on different GCLs?\n\n2. The original goal of prompt designing is to better elicit the knowledge contained in the pre-trained model for downstream applications. The second proposed method (CPTPP-M) for prompt generation actually obtains user and item embedding using adjacency matrix factorization separately which is irrelevant from GCL. Moreover, CPTPP-M performs the best half of the time. Can the authors explain the reason?\n\n3. Section 3.2.1, “As suggested in [15], the more uniform the embedding distribution is, the more capability to model the diverse preferences of users the method has”. Actually, this is a “speculation” in the paper [15] instead of a conclusion. Moreover, “uniform” distribution is quite ill-defined here. I find it hard to differentiate which one of the figures is more uniformly distributed. Are there any quantitative measurement methods?\n\n4. It seems that the proposed methods are not limited to user embeddings. Can the personalized prompt generation methods be symmetrically applied to items?\n", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed CPTPP reforms the existing GCL-based recommendation methods by separating the GCL pre-training and the downstream recommendation task using the prompt tuning mechanism. The idea is innovative for GCL based recommendation.\n \n2. Integrating prompts could better elicit the knowledge within the pre-trained user and item embeddings. The authors propose three different prompt generation methods, which can be applied to situations where users’ side information is not available.\n\n3. The writing is good, which is well organized and easy to read. Most experimental details are provided.\n", "weaknesses": "1. Generating prompts for users in recommender systems has been proposed in existing work. The difference mainly lies in that this paper addresses the GCL-based recommendation situation, where no side information of users is available.\n\n2. As an empirical study, some important details are not fully explained. For example, in section 2.2, the authors stated that “we can adopt various GCL learning methods …, to obtain high-quality user and item embeddings.”. Which GCL method did the authors actually adopt in their experiments? This is not claimed in the paper.\n\nI have some other questions about the experiments. Please refer to the “Questions” section.\n\n3. The Ablation Studies (section 3.2.3) actually provides horizontal comparisons among CPTPP-M, CPTPP-H, and CPTPP-R, instead of standard ablation studies. It is nice to provide some insights into the experiments. But I wonder to what extent the proposed methods improve the performance compared with models without the personalized prompts. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for recognising the contribution and novelty of our research work. We appreciate your suggestions and questions, and we will revise the manuscript according to your comments. The following is our response to your comments.\n\nW1. Generating prompts for users in recommender systems has been proposed in existing work. The difference mainly lies in that this paper addresses the GCL-based recommendation situation, where no side information of users is available.\n\nWe totally agree with you. Generating prompts in conventional recommender systems is a widely explored topic, focusing on utilising various side information of users and items. However, in our settings, we focus on the graph-based recommender system where only the user-item interaction graph is available, and no side information is provided. And the scope of our work is to adaptively construct prompts from the interaction graph to enhance the recommendation performance.\n\nW2. As an empirical study, some important details are not fully explained. For example, in section 2.2, the authors stated that “we can adopt various GCL learning methods …, to obtain high-quality user and item embeddings.”. Which GCL method did the authors actually adopt in their experiments? This is not claimed in the paper.\n\nWe apologise for neglecting the details about the base GCL model we adopt. To ensure that delicate GCL models do not cause performance improvement, we take SGL, a vanilla GCL model without sophisticated components, as the base model. SGL is the predecessor of the baseline SimGCL. We also strictly follow the evaluation protocols in GraphCL and InfoGraph to ensure a fair comparison. We will add related descriptions in the manuscript later. Thanks for your reminder.\n\nW3. The Ablation Studies (section 3.2.3) actually provides horizontal comparisons among CPTPP-M, CPTPP-H, and CPTPP-R, instead of standard ablation studies. It is nice to provide some insights into the experiments. But I wonder to what extent the proposed methods improve the performance compared with models without the personalized prompts.\n\nThanks for your suggestions. It is important to show the improvement brought by personalised prompts explicitly. Considering the base model we adopt is SGL, we will conduct experiments on SGL to compare it with our proposed method to verify the advantages of personalised prompts generation.\n\nQ1. There are three types in the baselines we selected. BPR-MF is a conventional recommendation method. BUIR and SelfCF are both contrastive learning-based recommendation systems. NCL and SimGCL are two representative GCL-based recommendation systems. As to our proposed method, we take SGL as the backbone, the predecessor version of SimGCL.\n\nThe generation modules in our proposed method have the same operations as our framework and can take various GCL-based recommendation methods as the backbone. The generation modules are independent of the GCL module. We can combine the generated prompts with the outputs of the GCL module to help to improve the recommendation performance.\n\nQ2. We agree that the original goal of prompt learning is to elicit the pre-trained model. However, it is time-consuming and requires experts while designing prompts. There are research works proposed a soft prompt paradigm [1][2] to address such limitations. Such a novel paradigm inspires our work. The fundamental procedure of soft prompts is to adaptively generate prompts based on side information like user profiles.\n\nThree user profiles are proposed in our work for graph-based recommendation scenarios without side information. All of them, including adjacency matrix factorisation (MF), are highly related to recommendation tasks. MF is a conventional recommendation method. It can produce high-quality low-dimension embeddings for users and items, reflecting the preferences of users and item features. So we can utilise the outputs of MF as the user profile for the downstream personalised prompt generation. Please note that the user profile acquisition is optional to be highly related to GCL.\n\nHistorical interaction records and high-order user relations are simple aggregation methods, combining user embeddings and items related to the target users. However, adjacency matrix factorisation is a machine learning method that can help embed initial user and item representations to a low-dimension latent space. A trainable embedding method can produce better user profiles than the two simple aggregation methods. That is a potential reason why CPTPP-M has better performance.\n\n[1] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts.\n[2] Yiqing Wu, Ruobing Xie, Yongchun Zhu, Fuzhen Zhuang, Xu Zhang, Leyu Lin, and Qing He. Personalized prompts for sequential recommendation.\n\nQ3. Thanks for your reminder. We acknowledge the limitations of our visualisations. Determining which is more uniformly distributed according to the visualisation results is not straightforward. There is a potential alternative measurement. We can first generate a 2-dimension uniform distribution. Then, we use metrics like Kullback-Leibler divergence and Wasserstein distance to measure the difference between the generated uniform distribution and the distribution of obtained user embeddings. A slight divergence or distance indicates a minor difference. In this way, we may offer a quantitative method to measure the \"uniformity\". We will add quantitative results together with the visualisation to demonstrate the quality of obtained user embeddings.\n\nQ4. Yes, it can be symmetrically applied to items. Should the participants in some scenarios request to recommend users for a specific item to construct the list of potential clients, they can do it symmetrically."}, {"review_id": "xLENvgzFlX", "reviewer": "Reviewer_UMuQ", "summary": "This paper proposes a prompt-tuning approach for GCL-based recommender systems. A framework consisting of a GCL module, a prompt generation module and a recommendation module is developed. Both ablation study and hyper-parameter study are conducted.", "questions": "See the weakness.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper studies an interesting research problem, prompt tuning for GCL-based recommendation, which gets rid of a combination of two quite different targets.\n- The paper is easy to follow.\n- Both ablation study and hyper-parameter study are conducted.", "weaknesses": "- The improvements over existing baselines are not significant. For example, SimGCL outperforms the proposed CPTPP approach on Gowalla dataset w.r.t NDCG@20.\n- The visualization of Figure 2 is not convincing to me.\n- There is no doc or README in the released codes.\n- The authors did not provide the reasons for selecting the adopted baselines.\n- What is the computational complexity of the proposed method compared with existing approaches?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We would like to respond to your comments and questions in the following.\n\nW1. The improvements over existing baselines are not significant. For example, SimGCL outperforms the proposed CPTPP approach on Gowalla dataset w.r.t NDCG@20.\n\nThanks for your conscientious and detailed review. We will keep tuning the proposed model. Currently, CPPTPP outperforms the baselines in most cases and has a very close performance compared to SimGCL on dataset Gowalla regarding the metric NDCG@20.\n\nW2. The visualization of Figure 2 is not convincing to me.\n\nThe reviewer PJh3 has the same concern regarding the visualisation results. Please refer to Q3 in the reponse to reviewer PJh3.\n\nW3. There is no doc or README in the released codes.\n\nThanks for your reminder. We will add the related document later. The document is also available in the SelfRec project on Github as I officially stated that our implementation is based on the source codes of SelfRec.\n\nW4. The authors did not provide the reasons for selecting the adopted baselines.\n\nThere are three types in the baselines we selected. BPR-MF is a conventional recommendation method. BUIR and SelfCF are both contrastive learning-based recommendation systems. NCL and SimGCL are two representative GCL-based recommendation systems. As to our proposed method, we take SGL as the backbone, the predecessor version of SimGCL. We will add related information in the main content later.\n\nW5. What is the computational complexity of the proposed method compared with existing approaches?\n\nOur proposed method is a framework to address the limitations of current GCL-based recommendation methods. The complexity of the proposed methods depends on the complexity of backbone GCL method and the user profile generation method. Therefore, there is no fixed complexity for our proposed method."}, {"review_id": "NkpKQWCfsI", "reviewer": "Reviewer_NDuk", "summary": "This paper proposes a prompt-enhanced framework for GCL-based recommender named CPTPP. At the core of CPTPP is a personalized user prompts generation framework that summarizes user profiles in graph recommender systems. The generated user prompts are then integrated with pre-trained user embeddings when applied in downstream tasks. Empirical results on three benchmark datasets shows that CPTPP is able to outperform state-of-the-art baseline alternatives such as SimGCL.", "questions": "Why does combining personalized user prompts with pre-trained user embedding help narrow the distinct targets between pre-training and downstream tasks? The paper explains in detail how the personalized user prompts are created, but does not explain why this results in narrowed distinction.", "limitations": "CPTPP relies on mining historical interaction records, adjacency matrix, and high-order user relations for generating personalized user prompts, and therefore it is possible that the improved performance primarily comes from the extra features used.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- CPTPP achieves strong empirical performance, outperforming strong baseline approaches such as SimGCL.\n- Graph contrastive pre-training is useful in the recommendation scenario with many potential downstream applications.\n- Code is available which improves reproducibility.", "weaknesses": "- CPTPP relies on exploiting historical interaction records, adjacency matrix, and high-order user relations for generating personalized user prompts. The personalized user prompts can thus be viewed as features engineered from user-item interactions. When incorporating feature engineering into the model, it is not surprising that there will be an improvement in terms of performance.\n- Performance is not much better even with the auxiliary information used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and questions. We will revise our paper according to them. Now, we would like to respond to your suggestions and questions.\n\nW1. CPTPP relies on exploiting historical interaction records, adjacency matrix, and high-order user relations for generating personalized user prompts. The personalized user prompts can thus be viewed as features engineered from user-item interactions. When incorporating feature engineering into the model, it is not surprising that there will be an improvement in terms of performance.\n\nOur method is inspired by a novel paradigm, soft prompt. The original goal of prompt learning is to elicit the pre-trained model. However, it is time-consuming and requires experts while designing hard prompts. The soft prompt paradigm addresses such limitation by utilising side information like user profiles to adaptively generate soft prompts. The generated prompts help the proposed method achieve better performance. Please notice that, the baselines also process the user-item interaction graphs but fail to outperform our method, which shows the advantages of our method to process the interaction graph, which is soft prompting.\n\nW2. Performance is not much better even with the auxiliary information used.\n\nPlease note that there is no auxiliary information available. The settings of our research is graph-based recommendation without side information. Only user-item interaction graph is available for each method. We first propose acquiring various user profiles based on user-item interaction graph and generating soft prompts based the acquired user profiles. No side information used in our method.\n\nQ1. Why does combining personalized user prompts with pre-trained user embedding help narrow the distinct targets between pre-training and downstream tasks? The paper explains in detail how the personalized user prompts are created, but does not explain why this results in narrowed distinction.\n\nOne of the advantages of prompt learning is to narrow the distinct targets between pre-training and downstream tasks, which is verified by many research works in the community of prompt learning. If the reviewer wants to know more about the mechanism and advantages of prompt learning, please refer to the section of related work, where we list some critical literature which is the basement of prompt learning.\n\nL1. CPTPP relies on mining historical interaction records, adjacency matrix, and high-order user relations for generating personalized user prompts, and therefore it is possible that the improved performance primarily comes from the extra features used.\n\nThanks for acknowledging that our proposed personalised prompts improve the performance. Please note that there is no extra features used in our method, the personalised prompts are generated solely based on the user-item interaction graph."}, {"review_id": "0pXpkkZRlp", "reviewer": "Reviewer_1hDm", "summary": "The paper presents an empirical study on the application of prompt-tuning for graph contrastive pre-training in recommendation systems. The authors propose a method that combines graph neural networks (GNNs) and contrastive learning to enhance the performance of recommendation models. The key idea is to leverage prompt engineering techniques, where carefully crafted prompts are used to guide the recommendation process.\n\nThe authors conduct extensive experiments on several real-world datasets, comparing their proposed method with various baselines. They evaluate the performance in terms of recommendation accuracy, coverage, and diversity. The results demonstrate the effectiveness of the proposed approach, showing significant improvements over the baselines in terms of recommendation quality.\n\nThe contributions of the paper include the introduction of a novel method that combines graph contrastive learning with prompt-tuning for recommendation systems. The authors provide insights into the design choices and hyper-parameter settings of the proposed method. They also conduct ablation studies to analyze the impact of different components and variations in the training process.\n\nOverall, the paper highlights the potential of prompt-tuning for graph contrastive pre-training in recommendation systems. The empirical results support the effectiveness of the proposed approach and provide valuable insights for researchers and practitioners in the field of recommendation systems.\n", "questions": "1. How sensitive is the proposed method to the choice of hyper-parameters, such as the prompt size or learning rates? Have you conducted sensitivity analysis to understand the impact of these hyper-parameters on the performance?\n2. How generalizable are the prompt engineering techniques used in this study? Can they be applied to other recommendation tasks or different prompt-based methods beyond the proposed approach?\n", "limitations": "Could you discuss potential challenges and limitations of prompt-tuning in recommendation systems? Are there any scenarios where prompt-tuning might not be suitable or effective?", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper explores the application of prompt-tuning techniques in the context of graph contrastive pre-training for recommendation systems. This novel research direction expands the understanding of prompt-based methods in the field of recommendation systems. The paper also offers insights into prompt design for graph contrastive pre-training in recommendation systems. The authors discuss the importance of considering domain knowledge and tailoring prompts to specific tasks, providing practical guidance for researchers and practitioners.\n\n2. The authors conduct systematic evaluations on different prompt strategies, considering both template-based prompts and prompt engineering. The evaluation process is well-designed, ensuring a comprehensive analysis of the effectiveness of prompt-tuning methods. Also, the paper provides a detailed comparative analysis of different prompt-tuning methods, allowing readers to understand the relative performance of each technique. This analysis helps researchers and practitioners make informed decisions regarding the choice of prompt strategy for graph contrastive pre-training.\n\n3. The experiments are conducted on various recommendation datasets, enhancing the generalizability of the findings. This inclusion of diverse datasets strengthens the validity of the conclusions drawn from the study. The authors also pay attention to hyperparameter tuning and conduct experiments to find optimal settings. This consideration enhances the reliability of the results and ensures that the performance improvements observed are not solely due to arbitrary hyperparameter choices.\n\n4. The paper includes detailed information about the code implementation and data availability, facilitating reproducibility and promoting further research in the field. This transparency enhances the credibility of the study.\n\n5. The findings of the paper have practical implications for the development of recommendation systems. The demonstrated performance improvements through prompt-tuning techniques can guide practitioners in enhancing the effectiveness of graph contrastive pre-training models for recommendation tasks. The paper's contributions have the potential to positively impact the field of recommendation systems. By introducing and validating the effectiveness of prompt-tuning for graph contrastive pre-training, the authors provide valuable insights that can guide future research and the development of improved recommendation algorithms.\n", "weaknesses": "1.The paper assumes readers have a strong understanding of graph contrastive learning and prompt-tuning, potentially alienating some readers unfamiliar with these topics.\n\n2. While the authors show the effectiveness of prompt-tuning, it would be interesting to compare their approach with prompt-less models to understand the benefits fully.\n\n3. The paper could discuss the generalizability of the prompt engineering techniques used in the study to other recommendation tasks or different prompt-based methods.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comprehensive reviewing and recognising our contributions. We sincerely appreciate your valuable comments and suggestions. We would response to your comments and concerns in the following.\n\nW1. The paper assumes readers have a strong understanding of graph contrastive learning and prompt-tuning, potentially alienating some readers unfamiliar with these topics.\n\nThanks for your reminder. We agree that our writing neglects the readers unfamiliar with the related topics and causes misunderstandings in other reviews. We will add some content as preliminaries to briefly introduce the background knowledge of GCL and prompt learning.\n\nW2. While the authors show the effectiveness of prompt-tuning, it would be interesting to compare their approach with prompt-less models to understand the benefits fully.\n\nYes, we agree with that. The reviewer PJh3 also has the same concern. It is important to show the improvement brought by personalised prompts explicitly. Considering the base model we adopt is SGL, we will conduct experiments on SGL to compare it with our proposed method to verify the advantages of personalised prompts generation.\n\nW3. The paper could discuss the generalizability of the prompt engineering techniques used in the study to other recommendation tasks or different prompt-based methods.\n\nThanks for your suggestions. We will collect more literature to discuss the application of prompt learning in other recommendation tasks. As the research scope of us focuses on graph learning, we apologies for limiting the discussion about it.\n\nQ1. How sensitive is the proposed method to the choice of hyper-parameters, such as the prompt size or learning rates? Have you conducted sensitivity analysis to understand the impact of these hyper-parameters on the performance?\n\nTo better understand the properties of our proposed CPTPP, we conduct hyper-parameter studies on an important term, the dimension size of the personalized prompt. By fixing all the other hyper-parameters, we comprehensively examine the performance of three versions of the proposed CPTPP on all the datasets with different prompts. Specifically, the size of the personalized prompt is selected from $\\{8, 16, 32, 64, 128, 256\\}$. We choose two metrics, Precision@5 and NDCG@5, to demonstrate CPTPP's performance variations with different prompt sizes. All the experiment results are shown in Figure 3 and Figure 5 in the Appendix.\n\n(i) The first thing we can observe is that, in most cases, CPTPP has the best performance when the prompt size is not larger than the dimensionality of user embeddings. A potential reason is that sizeable prompt dimensions would introduce more noise into pre-trained user embeddings, disturbing the structural semantics extracted from the user-item interaction graph by graph contrastive learning. \n(ii) We also notice a significant performance improvement when prompt size is 256 in several cases, such as CPTPP-M on dataset ML-1M and CPTPP-R on dataset Gowalla.\nHowever, they still fail to significantly outperform the CPTPP model, which has a much smaller prompt size. Therefore, small prompt size for prompt-tuning is a better option in practice as they achieve a relatively good recommendation quality and higher efficiency.\n\nQ2. How generalizable are the prompt engineering techniques used in this study? Can they be applied to other recommendation tasks or different prompt-based methods beyond the proposed approach?\n\nThe proposed method can be applied to various graph-based recommendation tasks and enhance the recommendation performance when no side information is available. Please note that the final outputs of our proposed method are the user embeddings enriched by the personalised prompts and item embeddings. The acquired user and item representations can be fed into any downstream tasks that accept such inputs. Therefore, the generalizability of our proposed method is promising.\n\nL1. Could you discuss potential challenges and limitations of prompt-tuning in recommendation systems? Are there any scenarios where prompt-tuning might not be suitable or effective?\n\nThe challenges of applying prompt learning are time-consuming and require experts knowledge to design the prompts. Though a novel paradigm, soft prompt, is proposed, it requires auxiliary information like user profiles and extra computation resources to generate prompts according to the auxiliary information. The prompt-tuning paradigm may not work when the computation source is limited (e.g., edge device). Moreover, the prompt-tuning is tailored for pre-trained models. If there is no pre-trained models get involved, the prompt-tuning cannot be applied."}]

Author rebuttal:

Meta-review: Overall, the reviewers all believe that this work attempts to address an important research problem and the proposed method has sufficient novelty. Although some reviewers raised questions about the adequacy of the experiments and the clarity of the description, the consensus in the discussion is that the authors would be able to address these issues well in their final version.
Decision: Accept (poster)